[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:02:12 dist-test-slave-69rg krb5kdc[9668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:02:12 dist-test-slave-69rg krb5kdc[9668](info): set up 2 sockets
Aug 06 17:02:12 dist-test-slave-69rg krb5kdc[9668](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 17:02:13.989446 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.008s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:02:14 dist-test-slave-69rg krb5kdc[9668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:02:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:02:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:02:14.131922 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:44801
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:44801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:14.255867 9683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:14.256235 9683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:14.256325 9683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:14.260581 9683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:14.260677 9683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:14.260717 9683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:14.260774 9683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:14.260811 9683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:02:14.266564 9683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:44801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:44801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:14.267799 9683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:14.268918 9683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:14.278513 9691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:14.278785 9683 server_base.cc:1034] running on GCE node
W20260806 17:02:14.278493 9689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:14.278491 9688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:14.279664 9683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:14.280926 9683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:14.282109 9683 hybrid_clock.cc:648] HybridClock initialized: now 1786035734282086 us; error 70 us; skew 500 ppm
Aug 06 17:02:14 dist-test-slave-69rg krb5kdc[9668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035734, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:14.285866 9683 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:02:14.286947 9683 webserver.cc:460] Webserver started at http://127.9.111.126:41389/ using document root <none> and password file <none>
I20260806 17:02:14.287529 9683 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:14.287595 9683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:14.287775 9683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:14.290153 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "7011ce09e7144d8f9c5843b6ea476748"
format_stamp: "Formatted at 2026-08-06 17:02:14 on dist-test-slave-69rg"
server_key: "3ed1a4c6f272f849d4ffa1d37186972e"
server_key_iv: "953ae25c20c0c2233dd13e42b284342b"
server_key_version: "encryptionkey@0"
I20260806 17:02:14.290748 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "7011ce09e7144d8f9c5843b6ea476748"
format_stamp: "Formatted at 2026-08-06 17:02:14 on dist-test-slave-69rg"
server_key: "3ed1a4c6f272f849d4ffa1d37186972e"
server_key_iv: "953ae25c20c0c2233dd13e42b284342b"
server_key_version: "encryptionkey@0"
I20260806 17:02:14.295013 9683 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 17:02:14.298046 9700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:14.299588 9683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 17:02:14.299782 9683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "7011ce09e7144d8f9c5843b6ea476748"
format_stamp: "Formatted at 2026-08-06 17:02:14 on dist-test-slave-69rg"
server_key: "3ed1a4c6f272f849d4ffa1d37186972e"
server_key_iv: "953ae25c20c0c2233dd13e42b284342b"
server_key_version: "encryptionkey@0"
I20260806 17:02:14.299930 9683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:14.317934 9683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:14.321755 9683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:14.322052 9683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:14.330610 9683 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:44801
I20260806 17:02:14.330626 9753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:44801 every 8 connection(s)
I20260806 17:02:14.331764 9683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:02:14.335454 9754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:14.341300 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 9683
I20260806 17:02:14.341527 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:02:14.343096 9754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748: Bootstrap starting.
I20260806 17:02:14.343489 9661 external_mini_cluster.cc:1428] Setting key 14fb8eecd858d263fed58bf95bacbd04
I20260806 17:02:14.346612 9754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:14.347496 9754 log.cc:824] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:14.350152 9754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748: No bootstrap required, opened a new log
I20260806 17:02:14.355212 9754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7011ce09e7144d8f9c5843b6ea476748" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44801 } }
I20260806 17:02:14.355526 9754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:14.355599 9754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7011ce09e7144d8f9c5843b6ea476748, State: Initialized, Role: FOLLOWER
I20260806 17:02:14.356135 9754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7011ce09e7144d8f9c5843b6ea476748" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44801 } }
I20260806 17:02:14.356362 9754 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:02:14.356432 9754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:02:14.356549 9754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:14.358075 9754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7011ce09e7144d8f9c5843b6ea476748" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44801 } }
I20260806 17:02:14.358568 9754 leader_election.cc:302] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7011ce09e7144d8f9c5843b6ea476748; no voters:
I20260806 17:02:14.358971 9754 leader_election.cc:288] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:02:14.359112 9759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:14.359390 9759 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748 [term 1 LEADER]: Becoming Leader. State: Replica: 7011ce09e7144d8f9c5843b6ea476748, State: Running, Role: LEADER
I20260806 17:02:14.359716 9759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7011ce09e7144d8f9c5843b6ea476748" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44801 } }
I20260806 17:02:14.360548 9754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:14.362053 9761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7011ce09e7144d8f9c5843b6ea476748. Latest consensus state: current_term: 1 leader_uuid: "7011ce09e7144d8f9c5843b6ea476748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7011ce09e7144d8f9c5843b6ea476748" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44801 } } }
I20260806 17:02:14.362221 9761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:14.362079 9760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7011ce09e7144d8f9c5843b6ea476748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7011ce09e7144d8f9c5843b6ea476748" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44801 } } }
I20260806 17:02:14.362490 9760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:14.363157 9768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:14.366799 9768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 17:02:14 dist-test-slave-69rg krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:14.373695 9768 catalog_manager.cc:1347] Generated new cluster ID: 08cb1eddea2b4209aa1542548756e697
I20260806 17:02:14.373776 9768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:14.378435 9756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:14.347373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.9.111.126:44801)
0806 17:02:14.347936 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:14.347957 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:14.349286 (+ 1329us) server_negotiation.cc:407] Connection header received
0806 17:02:14.354773 (+ 5487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:14.354805 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:14.355210 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:14.355599 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:14.359906 (+ 4307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:14.361933 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:14.364048 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:14.364558 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:14.364755 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:14.370442 (+ 5687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:14.370451 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:14.370469 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:14.370513 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:14.373320 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:14.374339 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:14.374345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:14.374351 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:14.374751 (+ 400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:14.375241 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:14.375244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:14.375246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:14.375897 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:14.376228 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:14.376952 (+ 724us) server_negotiation.cc:299] Negotiation successful
0806 17:02:14.377261 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":160,"threads_started":1}
I20260806 17:02:14.395465 9768 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:02:14.396886 9768 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:14.414139 9768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7011ce09e7144d8f9c5843b6ea476748: Generated new TSK 0
I20260806 17:02:14.415099 9768 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
I20260806 17:02:14.454176 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:44801
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:14.583674 9781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:14.583981 9781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:14.584031 9781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:14.588310 9781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:14.588399 9781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:14.588464 9781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:02:14.594085 9781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:44801
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:14.595284 9781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:14.596387 9781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:14.606302 9786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:14.606287 9787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:14.606294 9789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:14.606631 9781 server_base.cc:1034] running on GCE node
I20260806 17:02:14.607096 9781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:14.607769 9781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:14.608970 9781 hybrid_clock.cc:648] HybridClock initialized: now 1786035734608960 us; error 28 us; skew 500 ppm
Aug 06 17:02:14 dist-test-slave-69rg krb5kdc[9668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035734, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:14.611900 9781 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:02:14.612968 9781 webserver.cc:460] Webserver started at http://127.9.111.65:36533/ using document root <none> and password file <none>
I20260806 17:02:14.613555 9781 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:14.613648 9781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:14.613955 9781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:14.616101 9781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "85e582983c434c0796a864c71ec02368"
format_stamp: "Formatted at 2026-08-06 17:02:14 on dist-test-slave-69rg"
server_key: "0cfc5ca78625acc61697e0605c13afef"
server_key_iv: "fd215888875404e9ea17d23c8794e71b"
server_key_version: "encryptionkey@0"
I20260806 17:02:14.616695 9781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "85e582983c434c0796a864c71ec02368"
format_stamp: "Formatted at 2026-08-06 17:02:14 on dist-test-slave-69rg"
server_key: "0cfc5ca78625acc61697e0605c13afef"
server_key_iv: "fd215888875404e9ea17d23c8794e71b"
server_key_version: "encryptionkey@0"
I20260806 17:02:14.620496 9781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 17:02:14.623062 9796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:14.624310 9781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 17:02:14.624466 9781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "85e582983c434c0796a864c71ec02368"
format_stamp: "Formatted at 2026-08-06 17:02:14 on dist-test-slave-69rg"
server_key: "0cfc5ca78625acc61697e0605c13afef"
server_key_iv: "fd215888875404e9ea17d23c8794e71b"
server_key_version: "encryptionkey@0"
I20260806 17:02:14.624585 9781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:14.638240 9781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:14.642000 9781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:14.642239 9781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:14.642979 9781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:14.643873 9781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:14.643946 9781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:14.644019 9781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:14.644062 9781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:14.654062 9781 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:46527
I20260806 17:02:14.654096 9911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:46527 every 8 connection(s)
I20260806 17:02:14.654992 9781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:02:14.663208 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 9781
I20260806 17:02:14.663352 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:02:14.663683 9661 external_mini_cluster.cc:1428] Setting key 26d6768dac0f86ec3cbdca4a763985c5
Aug 06 17:02:14 dist-test-slave-69rg krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035734, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:14.671640 9756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:14.656820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:37281 (local address 127.9.111.126:44801)
0806 17:02:14.656982 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:14.656987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:14.657892 (+ 905us) server_negotiation.cc:407] Connection header received
0806 17:02:14.659270 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:14.659273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:14.659322 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:14.659396 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:14.660561 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:14.661342 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:14.662714 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:14.663127 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:14.663345 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:14.666826 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:14.666833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:14.666840 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:14.666876 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:14.668794 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:14.669646 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:14.669650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:14.669651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:14.669739 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:14.670211 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:14.670213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:14.670214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:14.670420 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:14.670499 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:14.671332 (+ 833us) server_negotiation.cc:299] Negotiation successful
0806 17:02:14.671497 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 17:02:14.672940 9914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:14.657236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:44801 (local address 127.9.111.65:37281)
0806 17:02:14.657690 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:14.657732 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:14.659064 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:14.659521 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:14.659527 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:14.659894 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:14.660379 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:14.660392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:14.661483 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:14.661486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:14.662575 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:14.662586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:14.663403 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:14.663410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:14.663597 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:14.664425 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:14.664443 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:14.666622 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:14.668957 (+ 2335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:14.668965 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:14.668993 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:14.669453 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:14.669879 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:14.669886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:14.669891 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:14.670098 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:14.670532 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:14.670542 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:14.670952 (+ 410us) client_negotiation.cc:769] Sending connection context
0806 17:02:14.671203 (+ 251us) client_negotiation.cc:240] Negotiation successful
0806 17:02:14.671649 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":107,"threads_started":1}
I20260806 17:02:14.674852 9912 heartbeater.cc:347] Connected to a master server at 127.9.111.126:44801
I20260806 17:02:14.675150 9912 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.111.66@KRBTEST.COM; defaulting to no policy
I20260806 17:02:14.675825 9912 heartbeater.cc:511] Master 127.9.111.126:44801 requested a full tablet report, sending...
I20260806 17:02:14.677520 9717 ts_manager.cc:194] Registered new tserver with Master: 85e582983c434c0796a864c71ec02368 (127.9.111.65:46527)
I20260806 17:02:14.678889 9717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:37281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
I20260806 17:02:14.697530 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:44801
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:14.821893 9917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:14.822218 9917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:14.822266 9917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:14.827104 9917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:14.827199 9917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:14.827270 9917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:02:14.833070 9917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:44801
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:14.834277 9917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:14.835283 9917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:14.845687 9923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:14.845767 9924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:14.845798 9926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:14.845937 9917 server_base.cc:1034] running on GCE node
I20260806 17:02:14.846714 9917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:14.847406 9917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:14.848618 9917 hybrid_clock.cc:648] HybridClock initialized: now 1786035734848611 us; error 32 us; skew 500 ppm
Aug 06 17:02:14 dist-test-slave-69rg krb5kdc[9668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035734, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:14.851584 9917 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:02:14.852609 9917 webserver.cc:460] Webserver started at http://127.9.111.66:42429/ using document root <none> and password file <none>
I20260806 17:02:14.853183 9917 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:14.853245 9917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:14.853420 9917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:14.855787 9917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "1db07144e2274cc9b118501a7951ac34"
format_stamp: "Formatted at 2026-08-06 17:02:14 on dist-test-slave-69rg"
server_key: "d016df9ab1a1f354f0e192434fe9d267"
server_key_iv: "607dbccab21265294c1fd57443161324"
server_key_version: "encryptionkey@0"
I20260806 17:02:14.856366 9917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "1db07144e2274cc9b118501a7951ac34"
format_stamp: "Formatted at 2026-08-06 17:02:14 on dist-test-slave-69rg"
server_key: "d016df9ab1a1f354f0e192434fe9d267"
server_key_iv: "607dbccab21265294c1fd57443161324"
server_key_version: "encryptionkey@0"
I20260806 17:02:14.861018 9917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260806 17:02:14.864192 9933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:14.865708 9917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 17:02:14.865916 9917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "1db07144e2274cc9b118501a7951ac34"
format_stamp: "Formatted at 2026-08-06 17:02:14 on dist-test-slave-69rg"
server_key: "d016df9ab1a1f354f0e192434fe9d267"
server_key_iv: "607dbccab21265294c1fd57443161324"
server_key_version: "encryptionkey@0"
I20260806 17:02:14.866058 9917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:14.877200 9917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:14.881191 9917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:14.881462 9917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:14.882220 9917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:14.883209 9917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:14.883288 9917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:14.883361 9917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:14.883414 9917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:14.896472 9917 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:38805
I20260806 17:02:14.896507 10049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:38805 every 8 connection(s)
I20260806 17:02:14.897487 9917 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:02:14.906248 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 9917
I20260806 17:02:14.906387 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:02:14.906766 9661 external_mini_cluster.cc:1428] Setting key fa3cf5b09b8bd97edacbb86965c3f84d
Aug 06 17:02:14 dist-test-slave-69rg krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035734, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:14.914427 9756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:14.899623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:35195 (local address 127.9.111.126:44801)
0806 17:02:14.899795 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:14.899800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:14.900712 (+ 912us) server_negotiation.cc:407] Connection header received
0806 17:02:14.902056 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:14.902059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:14.902111 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:14.902188 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:14.903533 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:14.904111 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:14.905495 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:14.905727 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:14.905898 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:14.909346 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:14.909353 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:14.909359 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:14.909406 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:14.911498 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:14.912363 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:14.912365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:14.912367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:14.912453 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:14.912938 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:14.912945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:14.912948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:14.913198 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:14.913294 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:14.914011 (+ 717us) server_negotiation.cc:299] Negotiation successful
0806 17:02:14.914170 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 17:02:14.915381 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:14.900052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:44801 (local address 127.9.111.66:35195)
0806 17:02:14.900545 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:14.900589 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:14.901771 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:14.902383 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:14.902390 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:14.902780 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:14.903379 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:14.903394 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:14.904292 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:14.904295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:14.905357 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:14.905367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:14.906045 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:14.906049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:14.906341 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:14.907172 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:14.907193 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:14.909119 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:14.911690 (+ 2571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:14.911701 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:14.911726 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:14.912187 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:14.912554 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:14.912559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:14.912562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:14.912786 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:14.913350 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:14.913358 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:14.913674 (+ 316us) client_negotiation.cc:769] Sending connection context
0806 17:02:14.913947 (+ 273us) client_negotiation.cc:240] Negotiation successful
0806 17:02:14.914295 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":116,"threads_started":1}
I20260806 17:02:14.917232 10050 heartbeater.cc:347] Connected to a master server at 127.9.111.126:44801
I20260806 17:02:14.917529 10050 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:14.918236 10050 heartbeater.cc:511] Master 127.9.111.126:44801 requested a full tablet report, sending...
I20260806 17:02:14.919756 9717 ts_manager.cc:194] Registered new tserver with Master: 1db07144e2274cc9b118501a7951ac34 (127.9.111.66:38805)
WARNING: no policy specified for kudu/127.9.111.67@KRBTEST.COM; defaulting to no policy
I20260806 17:02:14.920552 9717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:35195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
I20260806 17:02:14.942467 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:44801
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:15.065416 10055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:15.065735 10055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:15.065783 10055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:15.070163 10055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:15.070235 10055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:15.070298 10055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:02:15.076119 10055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:44801
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:15.077286 10055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:15.078440 10055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 17:02:15.089890 10055 server_base.cc:1034] running on GCE node
W20260806 17:02:15.089913 10064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:15.089795 10060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:15.089795 10061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:15.090649 10055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:15.091450 10055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:15.092697 10055 hybrid_clock.cc:648] HybridClock initialized: now 1786035735092666 us; error 42 us; skew 500 ppm
Aug 06 17:02:15 dist-test-slave-69rg krb5kdc[9668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035735, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:15.095991 10055 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:02:15.097203 10055 webserver.cc:460] Webserver started at http://127.9.111.67:40653/ using document root <none> and password file <none>
I20260806 17:02:15.097892 10055 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:15.097995 10055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:15.098269 10055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:15.100819 10055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "ea3c62db574c479cbb8b517f953dad3a"
format_stamp: "Formatted at 2026-08-06 17:02:15 on dist-test-slave-69rg"
server_key: "1c8f645cf1595f9d5c2f92b45c9a8b30"
server_key_iv: "6db6a394ba47fd188ccb64320c75680b"
server_key_version: "encryptionkey@0"
I20260806 17:02:15.101537 10055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "ea3c62db574c479cbb8b517f953dad3a"
format_stamp: "Formatted at 2026-08-06 17:02:15 on dist-test-slave-69rg"
server_key: "1c8f645cf1595f9d5c2f92b45c9a8b30"
server_key_iv: "6db6a394ba47fd188ccb64320c75680b"
server_key_version: "encryptionkey@0"
I20260806 17:02:15.106858 10055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260806 17:02:15.110064 10071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:15.111729 10055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 17:02:15.111896 10055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "ea3c62db574c479cbb8b517f953dad3a"
format_stamp: "Formatted at 2026-08-06 17:02:15 on dist-test-slave-69rg"
server_key: "1c8f645cf1595f9d5c2f92b45c9a8b30"
server_key_iv: "6db6a394ba47fd188ccb64320c75680b"
server_key_version: "encryptionkey@0"
I20260806 17:02:15.112036 10055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:15.130897 10055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:15.134541 10055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:15.134773 10055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:15.135422 10055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:15.136310 10055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:15.136359 10055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:15.136420 10055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:15.136453 10055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:15.149696 10055 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:45447
I20260806 17:02:15.149720 10188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:45447 every 8 connection(s)
I20260806 17:02:15.150722 10055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:02:15.151335 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 10055
I20260806 17:02:15.151474 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:02:15.151832 9661 external_mini_cluster.cc:1428] Setting key 36a54e76db7375b77605b89e76b0a11a
Aug 06 17:02:15 dist-test-slave-69rg krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035735, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:15.167783 9756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:15.152954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:46739 (local address 127.9.111.126:44801)
0806 17:02:15.153094 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:15.153101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:15.155059 (+ 1958us) server_negotiation.cc:407] Connection header received
0806 17:02:15.156079 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:15.156082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:15.156133 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:15.156209 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:15.157408 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:15.158069 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:15.159412 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:15.159652 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:15.159754 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:15.163048 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:15.163052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:15.163054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:15.163148 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:15.164810 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:15.165505 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:15.165509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:15.165510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:15.165605 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:15.166337 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:15.166339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:15.166340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:15.166621 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:15.166717 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:15.167457 (+ 740us) server_negotiation.cc:299] Negotiation successful
0806 17:02:15.167639 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 17:02:15.168824 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:15.153605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:44801 (local address 127.9.111.67:46739)
0806 17:02:15.154686 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:15.154752 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:15.155864 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:15.156377 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:15.156384 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:15.156763 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:15.157261 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:15.157276 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:15.158192 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:15.158195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:15.159240 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:15.159251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:15.159836 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:15.159839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:15.160000 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:15.160845 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:15.160863 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:15.162783 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:15.164953 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:15.164959 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:15.164974 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:15.165342 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:15.165837 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:15.165841 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:15.165843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:15.166033 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:15.166769 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:15.166779 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:15.167141 (+ 362us) client_negotiation.cc:769] Sending connection context
0806 17:02:15.167385 (+ 244us) client_negotiation.cc:240] Negotiation successful
0806 17:02:15.167924 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"thread_start_us":338,"threads_started":1}
I20260806 17:02:15.170420 10189 heartbeater.cc:347] Connected to a master server at 127.9.111.126:44801
I20260806 17:02:15.170734 10189 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:15.171305 10189 heartbeater.cc:511] Master 127.9.111.126:44801 requested a full tablet report, sending...
I20260806 17:02:15.172408 9717 ts_manager.cc:194] Registered new tserver with Master: ea3c62db574c479cbb8b517f953dad3a (127.9.111.67:45447)
I20260806 17:02:15.173038 9717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:46739
I20260806 17:02:15.180099 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 17:02:15 dist-test-slave-69rg krb5kdc[9668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035735, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 17:02:15 dist-test-slave-69rg krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035735, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.111.65@KRBTEST.COM
I20260806 17:02:15.211573 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:15.201376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.9.111.65:46527)
0806 17:02:15.201758 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:15.201766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:15.201792 (+ 26us) server_negotiation.cc:407] Connection header received
0806 17:02:15.202044 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:15.202051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:15.202248 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:15.202393 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:15.203203 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:15.204146 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:15.205010 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:15.205451 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:15.205540 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:15.208301 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:15.208311 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:15.208325 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:15.208351 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:15.209980 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:15.210388 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:15.210393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:15.210395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:15.210485 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:15.210728 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:15.210729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:15.210731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:15.210980 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:15.211184 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:15.211296 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 17:02:15.211428 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":114,"threads_started":1}
W20260806 17:02:15.212543 9824 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47608
Aug 06 17:02:15 dist-test-slave-69rg krb5kdc[9668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 17:02:15 dist-test-slave-69rg krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.65@KRBTEST.COM
I20260806 17:02:15.244738 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:15.235492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47614 (local address 127.9.111.65:46527)
0806 17:02:15.235686 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:15.235691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:15.235926 (+ 235us) server_negotiation.cc:407] Connection header received
0806 17:02:15.236054 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:15.236057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:15.236101 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:15.236175 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:15.236946 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:15.237456 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:15.238436 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:15.238689 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:15.238780 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:15.241279 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:15.241282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:15.241284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:15.241311 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:15.242955 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:15.243432 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:15.243437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:15.243439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:15.243528 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:15.243831 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:15.243835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:15.243836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:15.244019 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:15.244188 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:15.244427 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 17:02:15.244553 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 17:02:15.246353 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 9781
I20260806 17:02:15.253800 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 9917
I20260806 17:02:15.260174 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 10055
I20260806 17:02:15.267223 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 9683
2026-08-06T17:02:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2383 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:02:15 dist-test-slave-69rg krb5kdc[10203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:02:15 dist-test-slave-69rg krb5kdc[10203](info): set up 2 sockets
Aug 06 17:02:15 dist-test-slave-69rg krb5kdc[10203](info): commencing operation
krb5kdc: starting...
W20260806 17:02:16.368757 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:02:16 dist-test-slave-69rg krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:02:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:02:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:02:16.510879 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:36215
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:36215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:16.626107 10217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:16.626425 10217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:16.626469 10217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:16.630692 10217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:16.630765 10217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:16.630784 10217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:16.630802 10217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:16.630817 10217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:02:16.636253 10217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:36215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:36215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:16.637333 10217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:16.638499 10217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 17:02:16.647495 10217 server_base.cc:1034] running on GCE node
W20260806 17:02:16.647531 10225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:16.647542 10222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:16.647535 10223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:16.648510 10217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:16.649708 10217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:16.650873 10217 hybrid_clock.cc:648] HybridClock initialized: now 1786035736650845 us; error 45 us; skew 500 ppm
Aug 06 17:02:16 dist-test-slave-69rg krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035736, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:16.653972 10217 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:02:16.655051 10217 webserver.cc:460] Webserver started at http://127.9.111.126:45629/ using document root <none> and password file <none>
I20260806 17:02:16.655663 10217 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:16.655766 10217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:16.656001 10217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:16.658346 10217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "241dca1da0dd45b69d33579c7b2cbff2"
format_stamp: "Formatted at 2026-08-06 17:02:16 on dist-test-slave-69rg"
server_key: "c3e034f267bf252540378503b183cd15"
server_key_iv: "153f3cd612f847f97bfaf159de70514c"
server_key_version: "encryptionkey@0"
I20260806 17:02:16.659014 10217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "241dca1da0dd45b69d33579c7b2cbff2"
format_stamp: "Formatted at 2026-08-06 17:02:16 on dist-test-slave-69rg"
server_key: "c3e034f267bf252540378503b183cd15"
server_key_iv: "153f3cd612f847f97bfaf159de70514c"
server_key_version: "encryptionkey@0"
I20260806 17:02:16.663236 10217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 17:02:16.666038 10232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:16.667367 10217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 17:02:16.667529 10217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "241dca1da0dd45b69d33579c7b2cbff2"
format_stamp: "Formatted at 2026-08-06 17:02:16 on dist-test-slave-69rg"
server_key: "c3e034f267bf252540378503b183cd15"
server_key_iv: "153f3cd612f847f97bfaf159de70514c"
server_key_version: "encryptionkey@0"
I20260806 17:02:16.667665 10217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:16.682659 10217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:16.686208 10217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:16.686445 10217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:16.694545 10217 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:36215
I20260806 17:02:16.694574 10286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:36215 every 8 connection(s)
I20260806 17:02:16.695547 10217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:02:16.698665 10287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:16.698663 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 10217
I20260806 17:02:16.698817 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:02:16.699103 9661 external_mini_cluster.cc:1428] Setting key e9ca1ed84d950f0f6a1daf299ba9e73f
I20260806 17:02:16.705866 10287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2: Bootstrap starting.
Aug 06 17:02:16 dist-test-slave-69rg krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:16.708819 10287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:16.709750 10287 log.cc:824] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:16.712155 10287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2: No bootstrap required, opened a new log
I20260806 17:02:16.714452 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:16.700508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52324 (local address 127.9.111.126:36215)
0806 17:02:16.700942 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:16.700959 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:16.700996 (+ 37us) server_negotiation.cc:407] Connection header received
0806 17:02:16.701941 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:16.701962 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:16.702298 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:16.702580 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:16.703524 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:16.704510 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:16.705665 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:16.706247 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:16.706397 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:16.709264 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:16.709271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:16.709288 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:16.709322 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:16.711360 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:16.711909 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:16.711914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:16.711918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:16.712030 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:16.712340 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:16.712345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:16.712350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:16.712775 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:16.712913 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:16.713292 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 17:02:16.713659 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":142,"threads_started":1}
I20260806 17:02:16.716476 10287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241dca1da0dd45b69d33579c7b2cbff2" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36215 } }
I20260806 17:02:16.716761 10287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:16.716820 10287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 241dca1da0dd45b69d33579c7b2cbff2, State: Initialized, Role: FOLLOWER
I20260806 17:02:16.717216 10287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241dca1da0dd45b69d33579c7b2cbff2" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36215 } }
I20260806 17:02:16.717383 10287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:02:16.717468 10287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:02:16.717588 10287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:16.718703 10287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241dca1da0dd45b69d33579c7b2cbff2" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36215 } }
I20260806 17:02:16.719074 10287 leader_election.cc:302] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 241dca1da0dd45b69d33579c7b2cbff2; no voters:
I20260806 17:02:16.719444 10287 leader_election.cc:288] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:02:16.719585 10292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:16.719863 10292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2 [term 1 LEADER]: Becoming Leader. State: Replica: 241dca1da0dd45b69d33579c7b2cbff2, State: Running, Role: LEADER
I20260806 17:02:16.720194 10292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241dca1da0dd45b69d33579c7b2cbff2" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36215 } }
I20260806 17:02:16.720806 10287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:16.721977 10294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "241dca1da0dd45b69d33579c7b2cbff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241dca1da0dd45b69d33579c7b2cbff2" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36215 } } }
I20260806 17:02:16.722136 10294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:16.722432 10293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 241dca1da0dd45b69d33579c7b2cbff2. Latest consensus state: current_term: 1 leader_uuid: "241dca1da0dd45b69d33579c7b2cbff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241dca1da0dd45b69d33579c7b2cbff2" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36215 } } }
I20260806 17:02:16.722530 10301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:16.722558 10293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:16.725533 10301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:02:16.730782 10301 catalog_manager.cc:1347] Generated new cluster ID: 3acd85caa56244df9e3a16a8e8493ec0
I20260806 17:02:16.730863 10301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:16.746618 10301 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:02:16.747838 10301 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:16.751467 10301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 241dca1da0dd45b69d33579c7b2cbff2: Generated new TSK 0
I20260806 17:02:16.752174 10301 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
I20260806 17:02:16.798944 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:36215
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:16.925745 10313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:16.926097 10313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:16.926245 10313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:16.930358 10313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:16.930459 10313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:16.930555 10313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:02:16.936075 10313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:36215
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:16.937309 10313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:16.938463 10313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:16.947966 10319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:16.947966 10313 server_base.cc:1034] running on GCE node
W20260806 17:02:16.947999 10320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:16.947961 10322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:16.948771 10313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:16.949465 10313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:16.950644 10313 hybrid_clock.cc:648] HybridClock initialized: now 1786035736950608 us; error 54 us; skew 500 ppm
Aug 06 17:02:16 dist-test-slave-69rg krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035736, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:16.953671 10313 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:02:16.954660 10313 webserver.cc:460] Webserver started at http://127.9.111.65:35319/ using document root <none> and password file <none>
I20260806 17:02:16.955227 10313 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:16.955296 10313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:16.955538 10313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:16.957629 10313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "4c112b15f2d94ec4a4c53408d0497d21"
format_stamp: "Formatted at 2026-08-06 17:02:16 on dist-test-slave-69rg"
server_key: "6bc5eddacad9339a98cb8c95c999d30c"
server_key_iv: "f40d53bb4f654f2810bc7c24ca0b28a5"
server_key_version: "encryptionkey@0"
I20260806 17:02:16.958304 10313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "4c112b15f2d94ec4a4c53408d0497d21"
format_stamp: "Formatted at 2026-08-06 17:02:16 on dist-test-slave-69rg"
server_key: "6bc5eddacad9339a98cb8c95c999d30c"
server_key_iv: "f40d53bb4f654f2810bc7c24ca0b28a5"
server_key_version: "encryptionkey@0"
I20260806 17:02:16.962280 10313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:02:16.964886 10329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:16.966297 10313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:16.966449 10313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "4c112b15f2d94ec4a4c53408d0497d21"
format_stamp: "Formatted at 2026-08-06 17:02:16 on dist-test-slave-69rg"
server_key: "6bc5eddacad9339a98cb8c95c999d30c"
server_key_iv: "f40d53bb4f654f2810bc7c24ca0b28a5"
server_key_version: "encryptionkey@0"
I20260806 17:02:16.966578 10313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:16.989696 10313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:16.993177 10313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:16.993528 10313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:16.994251 10313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:16.995160 10313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:16.995249 10313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:16.995319 10313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:16.995370 10313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:17.005609 10313 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:44811
I20260806 17:02:17.005648 10448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:44811 every 8 connection(s)
I20260806 17:02:17.006716 10313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:02:17.007992 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 10313
I20260806 17:02:17.008234 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:02:17.008656 9661 external_mini_cluster.cc:1428] Setting key 41efc7f0e0f319b0b2e1a6bfe3b3f926
Aug 06 17:02:17 dist-test-slave-69rg krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035736, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.111.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.66@KRBTEST.COM" created.
I20260806 17:02:17.024117 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:17.009914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:36039 (local address 127.9.111.126:36215)
0806 17:02:17.010070 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:17.010075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:17.011246 (+ 1171us) server_negotiation.cc:407] Connection header received
0806 17:02:17.012412 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:17.012416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:17.012482 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:17.012569 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:17.014205 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.014755 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.016045 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.016310 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.016430 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:17.019630 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:17.019634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:17.019636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:17.019666 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:17.021386 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:17.022165 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:17.022169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:17.022171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:17.022267 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:17.022830 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:17.022833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:17.022835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:17.023051 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:17.023133 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:17.023860 (+ 727us) server_negotiation.cc:299] Negotiation successful
0806 17:02:17.023984 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 17:02:17.025372 10451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:17.010427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:36215 (local address 127.9.111.65:36039)
0806 17:02:17.011049 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:17.011100 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:17.012046 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:17.012942 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:17.012951 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:17.013412 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:17.014016 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:17.014035 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.014921 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.014929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:17.015879 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:17.015891 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.016497 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.016501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:17.016693 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:17.017381 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:17.017400 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:17.019403 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:17.021555 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:17.021562 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:17.021581 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:17.022042 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:17.022410 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:17.022416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:17.022417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:17.022703 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:17.023169 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:17.023176 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:17.023563 (+ 387us) client_negotiation.cc:769] Sending connection context
0806 17:02:17.023817 (+ 254us) client_negotiation.cc:240] Negotiation successful
0806 17:02:17.024230 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":179,"threads_started":1}
I20260806 17:02:17.026923 10449 heartbeater.cc:347] Connected to a master server at 127.9.111.126:36215
I20260806 17:02:17.027256 10449 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:17.027801 10449 heartbeater.cc:511] Master 127.9.111.126:36215 requested a full tablet report, sending...
I20260806 17:02:17.029443 10251 ts_manager.cc:194] Registered new tserver with Master: 4c112b15f2d94ec4a4c53408d0497d21 (127.9.111.65:44811)
I20260806 17:02:17.030697 10251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:36039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
I20260806 17:02:17.046533 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:36215
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:17.168766 10454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:17.169070 10454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:17.169117 10454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:17.173694 10454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:17.173811 10454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:17.173946 10454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:02:17.179517 10454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:36215
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:17.180621 10454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:17.181648 10454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:17.192219 10459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:17.192289 10460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:17.192219 10462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:17.192548 10454 server_base.cc:1034] running on GCE node
I20260806 17:02:17.193176 10454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:17.193917 10454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:17.195150 10454 hybrid_clock.cc:648] HybridClock initialized: now 1786035737195086 us; error 39 us; skew 500 ppm
Aug 06 17:02:17 dist-test-slave-69rg krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035737, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:17.198630 10454 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:02:17.199709 10454 webserver.cc:460] Webserver started at http://127.9.111.66:41709/ using document root <none> and password file <none>
I20260806 17:02:17.200335 10454 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:17.200439 10454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:17.200685 10454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:17.203052 10454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "7ac9b0bc86494e25b4b68d664699634d"
format_stamp: "Formatted at 2026-08-06 17:02:17 on dist-test-slave-69rg"
server_key: "cc747be94ff94bfd3781612abc66cb72"
server_key_iv: "3b6940214cc3602d27e634fe24567a7f"
server_key_version: "encryptionkey@0"
I20260806 17:02:17.203691 10454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "7ac9b0bc86494e25b4b68d664699634d"
format_stamp: "Formatted at 2026-08-06 17:02:17 on dist-test-slave-69rg"
server_key: "cc747be94ff94bfd3781612abc66cb72"
server_key_iv: "3b6940214cc3602d27e634fe24567a7f"
server_key_version: "encryptionkey@0"
I20260806 17:02:17.208091 10454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 17:02:17.210915 10469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:17.212436 10454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 17:02:17.212610 10454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "7ac9b0bc86494e25b4b68d664699634d"
format_stamp: "Formatted at 2026-08-06 17:02:17 on dist-test-slave-69rg"
server_key: "cc747be94ff94bfd3781612abc66cb72"
server_key_iv: "3b6940214cc3602d27e634fe24567a7f"
server_key_version: "encryptionkey@0"
I20260806 17:02:17.212790 10454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:17.236479 10454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:17.240319 10454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:17.240572 10454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:17.241297 10454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:17.242389 10454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:17.242471 10454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:17.242553 10454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:17.242602 10454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:17.253074 10454 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:38187
I20260806 17:02:17.253096 10586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:38187 every 8 connection(s)
I20260806 17:02:17.254248 10454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:02:17.255347 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 10454
I20260806 17:02:17.255450 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:02:17.255749 9661 external_mini_cluster.cc:1428] Setting key e65e51c365d361d71dab4b00964ce158
Aug 06 17:02:17 dist-test-slave-69rg krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035737, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.111.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.67@KRBTEST.COM" created.
I20260806 17:02:17.272980 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:17.256370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:40733 (local address 127.9.111.126:36215)
0806 17:02:17.256544 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:17.256549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:17.257573 (+ 1024us) server_negotiation.cc:407] Connection header received
0806 17:02:17.259327 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:17.259333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:17.259411 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:17.259521 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:17.261647 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.262305 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.264671 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.264935 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.265056 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:17.268562 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:17.268565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:17.268568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:17.268605 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:17.270317 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:17.270953 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:17.270957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:17.270959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:17.271075 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:17.271480 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:17.271484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:17.271486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:17.271711 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:17.271827 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:17.272640 (+ 813us) server_negotiation.cc:299] Negotiation successful
0806 17:02:17.272793 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 17:02:17.273931 10589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:17.256843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:36215 (local address 127.9.111.66:40733)
0806 17:02:17.257387 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:17.257448 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:17.259037 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:17.259877 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:17.259892 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:17.260557 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:17.261423 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:17.261452 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.262470 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.262478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:17.264334 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:17.264354 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.265164 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.265171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:17.265470 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:17.266303 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:17.266331 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:17.268295 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:17.270487 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:17.270494 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:17.270514 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:17.270833 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:17.271185 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:17.271189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:17.271191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:17.271375 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:17.271825 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:17.271833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:17.272328 (+ 495us) client_negotiation.cc:769] Sending connection context
0806 17:02:17.272571 (+ 243us) client_negotiation.cc:240] Negotiation successful
0806 17:02:17.272938 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":154,"threads_started":1}
I20260806 17:02:17.275261 10587 heartbeater.cc:347] Connected to a master server at 127.9.111.126:36215
I20260806 17:02:17.275565 10587 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:17.276190 10587 heartbeater.cc:511] Master 127.9.111.126:36215 requested a full tablet report, sending...
I20260806 17:02:17.277524 10251 ts_manager.cc:194] Registered new tserver with Master: 7ac9b0bc86494e25b4b68d664699634d (127.9.111.66:38187)
I20260806 17:02:17.278503 10251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:40733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
I20260806 17:02:17.294799 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:36215
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:17.424981 10592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:17.425318 10592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:17.425369 10592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:17.429672 10592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:17.429765 10592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:17.429857 10592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:02:17.435429 10592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:36215
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:17.436560 10592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:17.437600 10592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:17.448089 10598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:17.448130 10601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:17.448089 10599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:17.448469 10592 server_base.cc:1034] running on GCE node
I20260806 17:02:17.448930 10592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:17.449689 10592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:17.450948 10592 hybrid_clock.cc:648] HybridClock initialized: now 1786035737450926 us; error 28 us; skew 500 ppm
Aug 06 17:02:17 dist-test-slave-69rg krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035737, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:17.454139 10592 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:02:17.455088 10592 webserver.cc:460] Webserver started at http://127.9.111.67:40155/ using document root <none> and password file <none>
I20260806 17:02:17.455643 10592 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:17.455704 10592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:17.455879 10592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:17.458065 10592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "62bac345322e45b88028ba1c3584b7ab"
format_stamp: "Formatted at 2026-08-06 17:02:17 on dist-test-slave-69rg"
server_key: "7765e0688a71c17adb9f13852180a76d"
server_key_iv: "6d359c4b3303530d107468f628930256"
server_key_version: "encryptionkey@0"
I20260806 17:02:17.458662 10592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "62bac345322e45b88028ba1c3584b7ab"
format_stamp: "Formatted at 2026-08-06 17:02:17 on dist-test-slave-69rg"
server_key: "7765e0688a71c17adb9f13852180a76d"
server_key_iv: "6d359c4b3303530d107468f628930256"
server_key_version: "encryptionkey@0"
I20260806 17:02:17.462868 10592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 17:02:17.465420 10608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:17.466782 10592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 17:02:17.466889 10592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "62bac345322e45b88028ba1c3584b7ab"
format_stamp: "Formatted at 2026-08-06 17:02:17 on dist-test-slave-69rg"
server_key: "7765e0688a71c17adb9f13852180a76d"
server_key_iv: "6d359c4b3303530d107468f628930256"
server_key_version: "encryptionkey@0"
I20260806 17:02:17.467042 10592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:17.484681 10592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:17.488510 10592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:17.488750 10592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:17.489792 10592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:17.491328 10592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:17.491416 10592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:17.491473 10592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:17.491532 10592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:17.503544 10592 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:41087
I20260806 17:02:17.503582 10726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:41087 every 8 connection(s)
I20260806 17:02:17.504573 10592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:02:17.513940 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 10592
I20260806 17:02:17.514086 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:02:17.514446 9661 external_mini_cluster.cc:1428] Setting key 5d4fca42a05beb50f1b539af0baa8d47
Aug 06 17:02:17 dist-test-slave-69rg krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035737, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:17.519986 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:17.506828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:51117 (local address 127.9.111.126:36215)
0806 17:02:17.506984 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:17.506988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:17.507870 (+ 882us) server_negotiation.cc:407] Connection header received
0806 17:02:17.509169 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:17.509173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:17.509231 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:17.509313 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:17.510599 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.511153 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.512341 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.512570 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.512664 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:17.515716 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:17.515720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:17.515722 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:17.515748 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:17.517544 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:17.518246 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:17.518249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:17.518250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:17.518341 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:17.518699 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:17.518702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:17.518703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:17.518918 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:17.519000 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:17.519698 (+ 698us) server_negotiation.cc:299] Negotiation successful
0806 17:02:17.519836 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 17:02:17.520967 10729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:17.507229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:36215 (local address 127.9.111.67:51117)
0806 17:02:17.507706 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:17.507751 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:17.508938 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:17.509528 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:17.509535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:17.509932 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:17.510424 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:17.510438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.511272 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.511274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:17.512188 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:17.512196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.512718 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.512720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:17.512865 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:17.513505 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:17.513522 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:17.515538 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:17.517680 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:17.517686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:17.517704 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:17.518072 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:17.518433 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:17.518436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:17.518437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:17.518585 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:17.519022 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:17.519028 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:17.519332 (+ 304us) client_negotiation.cc:769] Sending connection context
0806 17:02:17.519500 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 17:02:17.519867 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":128,"threads_started":1}
I20260806 17:02:17.522405 10727 heartbeater.cc:347] Connected to a master server at 127.9.111.126:36215
I20260806 17:02:17.522663 10727 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:17.523250 10727 heartbeater.cc:511] Master 127.9.111.126:36215 requested a full tablet report, sending...
I20260806 17:02:17.524355 10251 ts_manager.cc:194] Registered new tserver with Master: 62bac345322e45b88028ba1c3584b7ab (127.9.111.67:41087)
I20260806 17:02:17.524955 10251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:51117
I20260806 17:02:17.528893 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 17:02:17.540506 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:17.532632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52332 (local address 127.9.111.126:36215)
0806 17:02:17.532799 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:17.532803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:17.532966 (+ 163us) server_negotiation.cc:407] Connection header received
0806 17:02:17.533074 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:17.533076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:17.533118 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:17.533193 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:17.533948 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.534494 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.535343 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.535569 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.535678 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:17.537170 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:17.537173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:17.537176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:17.537211 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:17.538685 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:17.539141 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:17.539149 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:17.539151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:17.539231 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:17.539530 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:17.539534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:17.539537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:17.539790 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:17.539938 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:17.540188 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 17:02:17.540334 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 17:02:17.545680 10251 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:02:17.549064 10251 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:02:17.564667 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:17.558150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:38187 (local address 127.0.0.1:39488)
0806 17:02:17.558614 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:17.558666 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:17.558805 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:17.559699 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:17.559706 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:17.559731 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:17.559910 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:17.559918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.561345 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.561349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:17.562760 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:17.562772 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.563922 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.563926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:17.564124 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:17.564190 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 17:02:17.564314 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 17:02:17.564459 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":3,"thread_start_us":116,"threads_started":1}
I20260806 17:02:17.564723 10740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:17.558269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39488 (local address 127.9.111.66:38187)
0806 17:02:17.559122 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:17.559135 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:17.559157 (+ 22us) server_negotiation.cc:407] Connection header received
0806 17:02:17.559233 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:17.559238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:17.559497 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:17.559699 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:17.560062 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.561190 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.562899 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.563781 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.563892 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:17.564050 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:17.564383 (+ 333us) server_negotiation.cc:299] Negotiation successful
0806 17:02:17.564484 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":135,"threads_started":1}
I20260806 17:02:17.565019 10741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:17.558975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:44811 (local address 127.0.0.1:35284)
0806 17:02:17.559992 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:17.560005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:17.560122 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:17.560554 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:17.560558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:17.560577 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:17.560702 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:17.560708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.562084 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.562090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:17.563342 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:17.563354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.564600 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.564603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:17.564701 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:17.564722 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 17:02:17.564807 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 17:02:17.564900 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"mutex_wait_us":28,"thread_start_us":91,"threads_started":1}
I20260806 17:02:17.565265 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:17.559042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.9.111.67:41087)
0806 17:02:17.559875 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:17.559888 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:17.559912 (+ 24us) server_negotiation.cc:407] Connection header received
0806 17:02:17.559991 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:17.559998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:17.560193 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:17.560350 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:17.560599 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.561841 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.563268 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.563936 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.564138 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:17.564285 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:17.564735 (+ 450us) server_negotiation.cc:299] Negotiation successful
0806 17:02:17.564989 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":128,"threads_started":1}
I20260806 17:02:17.565464 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:17.559155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.9.111.65:44811)
0806 17:02:17.559594 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:17.559601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:17.560105 (+ 504us) server_negotiation.cc:407] Connection header received
0806 17:02:17.560236 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:17.560240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:17.560422 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:17.560570 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:17.560823 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.561941 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.563519 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.564314 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.564409 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:17.564535 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:17.565201 (+ 666us) server_negotiation.cc:299] Negotiation successful
0806 17:02:17.565306 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":228,"threads_started":1}
I20260806 17:02:17.564666 10739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:17.558244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:41087 (local address 127.0.0.1:52784)
0806 17:02:17.559468 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:17.559484 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:17.559584 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:17.560339 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:17.560342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:17.560359 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:17.560458 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:17.560469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.561986 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.561990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:17.563131 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:17.563139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.564085 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.564089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:17.564252 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:17.564292 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 17:02:17.564379 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 17:02:17.564459 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1134,"mutex_wait_us":52,"thread_start_us":97,"threads_started":1}
I20260806 17:02:17.568032 10379 tablet_service.cc:1467] Processing CreateTablet for tablet 857fe0fe686f45119065d3533b0beaf2 (DEFAULT_TABLE table=test-table [id=0e9888c4bb204023847b5a857a70ec4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:17.568261 10656 tablet_service.cc:1467] Processing CreateTablet for tablet 857fe0fe686f45119065d3533b0beaf2 (DEFAULT_TABLE table=test-table [id=0e9888c4bb204023847b5a857a70ec4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:17.568259 10518 tablet_service.cc:1467] Processing CreateTablet for tablet 857fe0fe686f45119065d3533b0beaf2 (DEFAULT_TABLE table=test-table [id=0e9888c4bb204023847b5a857a70ec4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:17.569144 10379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857fe0fe686f45119065d3533b0beaf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:17.569355 10656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857fe0fe686f45119065d3533b0beaf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:17.569355 10518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857fe0fe686f45119065d3533b0beaf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:17.576402 10744 tablet_bootstrap.cc:492] T 857fe0fe686f45119065d3533b0beaf2 P 4c112b15f2d94ec4a4c53408d0497d21: Bootstrap starting.
I20260806 17:02:17.578323 10745 tablet_bootstrap.cc:492] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d: Bootstrap starting.
I20260806 17:02:17.578950 10744 tablet_bootstrap.cc:654] T 857fe0fe686f45119065d3533b0beaf2 P 4c112b15f2d94ec4a4c53408d0497d21: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:17.579845 10746 tablet_bootstrap.cc:492] T 857fe0fe686f45119065d3533b0beaf2 P 62bac345322e45b88028ba1c3584b7ab: Bootstrap starting.
I20260806 17:02:17.579886 10744 log.cc:824] T 857fe0fe686f45119065d3533b0beaf2 P 4c112b15f2d94ec4a4c53408d0497d21: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:17.581117 10745 tablet_bootstrap.cc:654] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:17.582158 10745 log.cc:824] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:17.582383 10746 tablet_bootstrap.cc:654] T 857fe0fe686f45119065d3533b0beaf2 P 62bac345322e45b88028ba1c3584b7ab: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:17.583097 10744 tablet_bootstrap.cc:492] T 857fe0fe686f45119065d3533b0beaf2 P 4c112b15f2d94ec4a4c53408d0497d21: No bootstrap required, opened a new log
I20260806 17:02:17.583405 10744 ts_tablet_manager.cc:1397] T 857fe0fe686f45119065d3533b0beaf2 P 4c112b15f2d94ec4a4c53408d0497d21: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 17:02:17.583993 10746 log.cc:824] T 857fe0fe686f45119065d3533b0beaf2 P 62bac345322e45b88028ba1c3584b7ab: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:17.584048 10745 tablet_bootstrap.cc:492] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d: No bootstrap required, opened a new log
I20260806 17:02:17.584244 10745 ts_tablet_manager.cc:1397] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 17:02:17.586401 10746 tablet_bootstrap.cc:492] T 857fe0fe686f45119065d3533b0beaf2 P 62bac345322e45b88028ba1c3584b7ab: No bootstrap required, opened a new log
I20260806 17:02:17.586583 10746 ts_tablet_manager.cc:1397] T 857fe0fe686f45119065d3533b0beaf2 P 62bac345322e45b88028ba1c3584b7ab: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 17:02:17.588136 10745 raft_consensus.cc:348] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac9b0bc86494e25b4b68d664699634d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 38187 } } peers { permanent_uuid: "62bac345322e45b88028ba1c3584b7ab" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41087 } } peers { permanent_uuid: "4c112b15f2d94ec4a4c53408d0497d21" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 44811 } }
I20260806 17:02:17.588455 10745 raft_consensus.cc:374] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:17.588515 10745 raft_consensus.cc:729] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac9b0bc86494e25b4b68d664699634d, State: Initialized, Role: FOLLOWER
I20260806 17:02:17.588999 10745 consensus_queue.cc:260] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac9b0bc86494e25b4b68d664699634d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 38187 } } peers { permanent_uuid: "62bac345322e45b88028ba1c3584b7ab" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41087 } } peers { permanent_uuid: "4c112b15f2d94ec4a4c53408d0497d21" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 44811 } }
I20260806 17:02:17.589382 10744 raft_consensus.cc:348] T 857fe0fe686f45119065d3533b0beaf2 P 4c112b15f2d94ec4a4c53408d0497d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac9b0bc86494e25b4b68d664699634d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 38187 } } peers { permanent_uuid: "62bac345322e45b88028ba1c3584b7ab" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41087 } } peers { permanent_uuid: "4c112b15f2d94ec4a4c53408d0497d21" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 44811 } }
I20260806 17:02:17.589666 10744 raft_consensus.cc:374] T 857fe0fe686f45119065d3533b0beaf2 P 4c112b15f2d94ec4a4c53408d0497d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:17.589730 10744 raft_consensus.cc:729] T 857fe0fe686f45119065d3533b0beaf2 P 4c112b15f2d94ec4a4c53408d0497d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c112b15f2d94ec4a4c53408d0497d21, State: Initialized, Role: FOLLOWER
I20260806 17:02:17.590148 10745 ts_tablet_manager.cc:1428] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 17:02:17.590202 10744 consensus_queue.cc:260] T 857fe0fe686f45119065d3533b0beaf2 P 4c112b15f2d94ec4a4c53408d0497d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac9b0bc86494e25b4b68d664699634d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 38187 } } peers { permanent_uuid: "62bac345322e45b88028ba1c3584b7ab" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41087 } } peers { permanent_uuid: "4c112b15f2d94ec4a4c53408d0497d21" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 44811 } }
I20260806 17:02:17.590556 10587 heartbeater.cc:503] Master 127.9.111.126:36215 was elected leader, sending a full tablet report...
I20260806 17:02:17.591051 10449 heartbeater.cc:503] Master 127.9.111.126:36215 was elected leader, sending a full tablet report...
I20260806 17:02:17.591233 10744 ts_tablet_manager.cc:1428] T 857fe0fe686f45119065d3533b0beaf2 P 4c112b15f2d94ec4a4c53408d0497d21: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260806 17:02:17.591467 10746 raft_consensus.cc:348] T 857fe0fe686f45119065d3533b0beaf2 P 62bac345322e45b88028ba1c3584b7ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac9b0bc86494e25b4b68d664699634d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 38187 } } peers { permanent_uuid: "62bac345322e45b88028ba1c3584b7ab" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41087 } } peers { permanent_uuid: "4c112b15f2d94ec4a4c53408d0497d21" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 44811 } }
I20260806 17:02:17.591745 10746 raft_consensus.cc:374] T 857fe0fe686f45119065d3533b0beaf2 P 62bac345322e45b88028ba1c3584b7ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:17.591818 10746 raft_consensus.cc:729] T 857fe0fe686f45119065d3533b0beaf2 P 62bac345322e45b88028ba1c3584b7ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62bac345322e45b88028ba1c3584b7ab, State: Initialized, Role: FOLLOWER
I20260806 17:02:17.592227 10746 consensus_queue.cc:260] T 857fe0fe686f45119065d3533b0beaf2 P 62bac345322e45b88028ba1c3584b7ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac9b0bc86494e25b4b68d664699634d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 38187 } } peers { permanent_uuid: "62bac345322e45b88028ba1c3584b7ab" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41087 } } peers { permanent_uuid: "4c112b15f2d94ec4a4c53408d0497d21" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 44811 } }
I20260806 17:02:17.593039 10727 heartbeater.cc:503] Master 127.9.111.126:36215 was elected leader, sending a full tablet report...
I20260806 17:02:17.593456 10746 ts_tablet_manager.cc:1428] T 857fe0fe686f45119065d3533b0beaf2 P 62bac345322e45b88028ba1c3584b7ab: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
W20260806 17:02:17.755599 10588 tablet.cc:2369] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 17:02:17.755766 10728 tablet.cc:2369] T 857fe0fe686f45119065d3533b0beaf2 P 62bac345322e45b88028ba1c3584b7ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 17:02:17.758423 10450 tablet.cc:2369] T 857fe0fe686f45119065d3533b0beaf2 P 4c112b15f2d94ec4a4c53408d0497d21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:02:17.942502 10750 raft_consensus.cc:482] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:17.942776 10750 raft_consensus.cc:504] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac9b0bc86494e25b4b68d664699634d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 38187 } } peers { permanent_uuid: "62bac345322e45b88028ba1c3584b7ab" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41087 } } peers { permanent_uuid: "4c112b15f2d94ec4a4c53408d0497d21" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 44811 } }
I20260806 17:02:17.943990 10750 leader_election.cc:288] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62bac345322e45b88028ba1c3584b7ab (127.9.111.67:41087), 4c112b15f2d94ec4a4c53408d0497d21 (127.9.111.65:44811)
I20260806 17:02:17.948561 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:17.944546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:52719 (local address 127.9.111.67:41087)
0806 17:02:17.944732 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:17.944740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:17.944925 (+ 185us) server_negotiation.cc:407] Connection header received
0806 17:02:17.945065 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:17.945068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:17.945125 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:17.945214 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:17.945605 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.946291 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.947495 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.947942 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.948084 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:17.948147 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:17.948337 (+ 190us) server_negotiation.cc:299] Negotiation successful
0806 17:02:17.948415 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 17:02:17.948585 10753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:17.944366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:41087 (local address 127.9.111.66:52719)
0806 17:02:17.944800 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:17.944816 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:17.944950 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:17.945231 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:17.945234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:17.945251 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:17.945450 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:17.945458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.946421 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.946424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:17.947360 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:17.947370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.948084 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.948088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:17.948206 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:17.948228 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 17:02:17.948320 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 17:02:17.948419 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":3,"thread_start_us":220,"threads_started":1}
I20260806 17:02:17.948606 10754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:17.944366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:44811 (local address 127.9.111.66:60507)
0806 17:02:17.944964 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:17.944979 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:17.945102 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:17.945399 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:17.945402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:17.945416 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:17.945678 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:17.945687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.946489 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.946495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:17.947417 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:17.947426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.948119 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.948123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:17.948243 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:17.948261 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 17:02:17.948351 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 17:02:17.948470 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":93,"thread_start_us":121,"threads_started":1}
I20260806 17:02:17.948588 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:17.944610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:60507 (local address 127.9.111.65:44811)
0806 17:02:17.944797 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:17.944802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:17.945080 (+ 278us) server_negotiation.cc:407] Connection header received
0806 17:02:17.945228 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:17.945231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:17.945281 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:17.945361 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:17.945806 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.946371 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.947554 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:17.947997 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:17.948133 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:17.948189 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:17.948369 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 17:02:17.948456 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 17:02:17.949317 10402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857fe0fe686f45119065d3533b0beaf2" candidate_uuid: "7ac9b0bc86494e25b4b68d664699634d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c112b15f2d94ec4a4c53408d0497d21" is_pre_election: true
I20260806 17:02:17.949431 10678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857fe0fe686f45119065d3533b0beaf2" candidate_uuid: "7ac9b0bc86494e25b4b68d664699634d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62bac345322e45b88028ba1c3584b7ab" is_pre_election: true
I20260806 17:02:17.949673 10678 raft_consensus.cc:2454] T 857fe0fe686f45119065d3533b0beaf2 P 62bac345322e45b88028ba1c3584b7ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac9b0bc86494e25b4b68d664699634d in term 0.
I20260806 17:02:17.949674 10402 raft_consensus.cc:2454] T 857fe0fe686f45119065d3533b0beaf2 P 4c112b15f2d94ec4a4c53408d0497d21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac9b0bc86494e25b4b68d664699634d in term 0.
I20260806 17:02:17.950243 10470 leader_election.cc:302] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4c112b15f2d94ec4a4c53408d0497d21, 62bac345322e45b88028ba1c3584b7ab, 7ac9b0bc86494e25b4b68d664699634d; no voters:
I20260806 17:02:17.950537 10750 raft_consensus.cc:2781] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:02:17.950644 10750 raft_consensus.cc:482] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:02:17.950712 10750 raft_consensus.cc:3037] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:17.951953 10750 raft_consensus.cc:504] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac9b0bc86494e25b4b68d664699634d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 38187 } } peers { permanent_uuid: "62bac345322e45b88028ba1c3584b7ab" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41087 } } peers { permanent_uuid: "4c112b15f2d94ec4a4c53408d0497d21" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 44811 } }
I20260806 17:02:17.952350 10750 leader_election.cc:288] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d [CANDIDATE]: Term 1 election: Requested vote from peers 62bac345322e45b88028ba1c3584b7ab (127.9.111.67:41087), 4c112b15f2d94ec4a4c53408d0497d21 (127.9.111.65:44811)
I20260806 17:02:17.952818 10402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857fe0fe686f45119065d3533b0beaf2" candidate_uuid: "7ac9b0bc86494e25b4b68d664699634d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c112b15f2d94ec4a4c53408d0497d21"
I20260806 17:02:17.952844 10678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857fe0fe686f45119065d3533b0beaf2" candidate_uuid: "7ac9b0bc86494e25b4b68d664699634d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62bac345322e45b88028ba1c3584b7ab"
I20260806 17:02:17.952986 10402 raft_consensus.cc:3037] T 857fe0fe686f45119065d3533b0beaf2 P 4c112b15f2d94ec4a4c53408d0497d21 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:17.952986 10678 raft_consensus.cc:3037] T 857fe0fe686f45119065d3533b0beaf2 P 62bac345322e45b88028ba1c3584b7ab [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:17.954571 10402 raft_consensus.cc:2454] T 857fe0fe686f45119065d3533b0beaf2 P 4c112b15f2d94ec4a4c53408d0497d21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac9b0bc86494e25b4b68d664699634d in term 1.
I20260806 17:02:17.954571 10678 raft_consensus.cc:2454] T 857fe0fe686f45119065d3533b0beaf2 P 62bac345322e45b88028ba1c3584b7ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac9b0bc86494e25b4b68d664699634d in term 1.
I20260806 17:02:17.954973 10470 leader_election.cc:302] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4c112b15f2d94ec4a4c53408d0497d21, 62bac345322e45b88028ba1c3584b7ab, 7ac9b0bc86494e25b4b68d664699634d; no voters:
I20260806 17:02:17.955143 10750 raft_consensus.cc:2781] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:17.955437 10750 raft_consensus.cc:686] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d [term 1 LEADER]: Becoming Leader. State: Replica: 7ac9b0bc86494e25b4b68d664699634d, State: Running, Role: LEADER
I20260806 17:02:17.955796 10750 consensus_queue.cc:237] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac9b0bc86494e25b4b68d664699634d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 38187 } } peers { permanent_uuid: "62bac345322e45b88028ba1c3584b7ab" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41087 } } peers { permanent_uuid: "4c112b15f2d94ec4a4c53408d0497d21" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 44811 } }
I20260806 17:02:17.959371 10250 catalog_manager.cc:5495] T 857fe0fe686f45119065d3533b0beaf2 P 7ac9b0bc86494e25b4b68d664699634d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ac9b0bc86494e25b4b68d664699634d (127.9.111.66). New cstate: current_term: 1 leader_uuid: "7ac9b0bc86494e25b4b68d664699634d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac9b0bc86494e25b4b68d664699634d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 38187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62bac345322e45b88028ba1c3584b7ab" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c112b15f2d94ec4a4c53408d0497d21" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 44811 } health_report { overall_health: UNKNOWN } } }
Aug 06 17:02:18 dist-test-slave-69rg krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.65@KRBTEST.COM
I20260806 17:02:18.033634 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:18.014577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35300 (local address 127.9.111.65:44811)
0806 17:02:18.014743 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:18.014748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:18.014992 (+ 244us) server_negotiation.cc:407] Connection header received
0806 17:02:18.015123 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:18.015126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:18.015180 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:18.015257 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:18.016429 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:18.017033 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:18.018738 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:18.019008 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:18.019133 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:18.029889 (+ 10756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:18.029909 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:18.029934 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:18.029976 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:18.031856 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:18.032412 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:18.032417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:18.032419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:18.032506 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:18.032766 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:18.032768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:18.032769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:18.033028 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:18.033213 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:18.033342 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 17:02:18.033448 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Aug 06 17:02:18 dist-test-slave-69rg krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035738, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 17:02:18 dist-test-slave-69rg krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035738, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.111.65@KRBTEST.COM
I20260806 17:02:18.067016 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:18.054877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.9.111.65:44811)
0806 17:02:18.055058 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:18.055064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:18.055230 (+ 166us) server_negotiation.cc:407] Connection header received
0806 17:02:18.055459 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:18.055462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:18.055507 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:18.055600 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:18.056443 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:18.057055 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:18.057981 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:18.058238 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:18.058357 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:18.062177 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:18.062181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:18.062183 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:18.062217 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:18.064466 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:18.065068 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:18.065073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:18.065074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:18.065278 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:18.065644 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:18.065650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:18.065652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:18.065950 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:18.066122 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:18.066715 (+ 593us) server_negotiation.cc:299] Negotiation successful
0806 17:02:18.066886 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Aug 06 17:02:18 dist-test-slave-69rg krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 17:02:18 dist-test-slave-69rg krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.65@KRBTEST.COM
I20260806 17:02:18.100541 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:18.090446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35314 (local address 127.9.111.65:44811)
0806 17:02:18.090649 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:18.090653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:18.090788 (+ 135us) server_negotiation.cc:407] Connection header received
0806 17:02:18.090947 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:18.090950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:18.090998 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:18.091063 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:18.091928 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:18.092463 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:18.093453 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:18.093761 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:18.093961 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:18.096672 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:18.096675 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:18.096677 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:18.096705 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:18.098303 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:18.098832 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:18.098836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:18.098837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:18.098945 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:18.099392 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:18.099395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:18.099396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:18.099601 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:18.099705 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:18.100060 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 17:02:18.100280 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260806 17:02:18.102178 10358 tablet_service.cc:2908] Rejecting scan request for tablet 857fe0fe686f45119065d3533b0beaf2: Uninitialized: clean time has not yet been initialized
I20260806 17:02:18.104318 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 10313
I20260806 17:02:18.111536 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 10454
I20260806 17:02:18.117914 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 10592
I20260806 17:02:18.124867 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 10217
2026-08-06T17:02:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2858 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:02:18 dist-test-slave-69rg krb5kdc[10769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:02:18 dist-test-slave-69rg krb5kdc[10769](info): set up 2 sockets
Aug 06 17:02:18 dist-test-slave-69rg krb5kdc[10769](info): commencing operation
krb5kdc: starting...
W20260806 17:02:19.208995 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:02:19 dist-test-slave-69rg krb5kdc[10769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:02:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:02:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:02:19.353754 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:46561
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:46561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:19.471951 10782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:19.472327 10782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:19.472420 10782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:19.476709 10782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:19.476825 10782 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 17:02:19.476856 10782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:19.476902 10782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:19.476958 10782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:19.476997 10782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:02:19.482713 10782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:46561
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:46561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:19.483893 10782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:19.484974 10782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:19.494167 10790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:19.494246 10788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:19.494204 10787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:19.494341 10782 server_base.cc:1034] running on GCE node
I20260806 17:02:19.495182 10782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:19.496441 10782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:19.497623 10782 hybrid_clock.cc:648] HybridClock initialized: now 1786035739497604 us; error 36 us; skew 500 ppm
Aug 06 17:02:19 dist-test-slave-69rg krb5kdc[10769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035739, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:19.500923 10782 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:02:19.502214 10782 webserver.cc:460] Webserver started at http://127.9.111.126:46029/ using document root <none> and password file <none>
I20260806 17:02:19.502887 10782 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:19.503001 10782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:19.503283 10782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:19.505736 10782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "ba7035a4ce364e81b797d4307edf3133"
format_stamp: "Formatted at 2026-08-06 17:02:19 on dist-test-slave-69rg"
server_key: "ca5989b7ee53131b6a8a97201285147c"
server_key_iv: "ff8d26a6beb43010eb2e26395d59db16"
server_key_version: "encryptionkey@0"
I20260806 17:02:19.506587 10782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "ba7035a4ce364e81b797d4307edf3133"
format_stamp: "Formatted at 2026-08-06 17:02:19 on dist-test-slave-69rg"
server_key: "ca5989b7ee53131b6a8a97201285147c"
server_key_iv: "ff8d26a6beb43010eb2e26395d59db16"
server_key_version: "encryptionkey@0"
I20260806 17:02:19.511101 10782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:02:19.514005 10797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:19.515471 10782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:19.515666 10782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "ba7035a4ce364e81b797d4307edf3133"
format_stamp: "Formatted at 2026-08-06 17:02:19 on dist-test-slave-69rg"
server_key: "ca5989b7ee53131b6a8a97201285147c"
server_key_iv: "ff8d26a6beb43010eb2e26395d59db16"
server_key_version: "encryptionkey@0"
I20260806 17:02:19.515801 10782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:19.530963 10782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:19.534394 10782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:19.534622 10782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:19.543808 10782 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:46561
I20260806 17:02:19.543881 10861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:46561 every 8 connection(s)
I20260806 17:02:19.544929 10782 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:02:19.547943 10862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:19.551899 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 10782
I20260806 17:02:19.552037 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:02:19.552307 9661 external_mini_cluster.cc:1428] Setting key e073a39dc479393140a0bd0a38af3e56
I20260806 17:02:19.554699 10862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133: Bootstrap starting.
I20260806 17:02:19.557096 10862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:19.557950 10862 log.cc:824] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:19.560135 10862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133: No bootstrap required, opened a new log
Aug 06 17:02:19 dist-test-slave-69rg krb5kdc[10769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:19.564158 10862 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7035a4ce364e81b797d4307edf3133" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 46561 } }
I20260806 17:02:19.564435 10862 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:19.564499 10862 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba7035a4ce364e81b797d4307edf3133, State: Initialized, Role: FOLLOWER
I20260806 17:02:19.564990 10862 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7035a4ce364e81b797d4307edf3133" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 46561 } }
I20260806 17:02:19.565128 10862 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:02:19.565250 10862 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:02:19.565390 10862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:19.566480 10862 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7035a4ce364e81b797d4307edf3133" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 46561 } }
I20260806 17:02:19.566843 10862 leader_election.cc:302] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba7035a4ce364e81b797d4307edf3133; no voters:
I20260806 17:02:19.567207 10862 leader_election.cc:288] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:02:19.567329 10867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:19.567574 10867 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133 [term 1 LEADER]: Becoming Leader. State: Replica: ba7035a4ce364e81b797d4307edf3133, State: Running, Role: LEADER
I20260806 17:02:19.567922 10867 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7035a4ce364e81b797d4307edf3133" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 46561 } }
I20260806 17:02:19.568619 10862 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:19.568951 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:19.553967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.9.111.126:46561)
0806 17:02:19.554492 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:19.554509 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:19.554548 (+ 39us) server_negotiation.cc:407] Connection header received
0806 17:02:19.555315 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:19.555337 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:19.555697 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:19.556067 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:19.556946 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:19.558182 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:19.559300 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:19.559800 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:19.559982 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:19.562941 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:19.562947 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:19.562964 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:19.563000 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:19.565112 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:19.565755 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:19.565764 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:19.565775 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:19.565980 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:19.566304 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:19.566308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:19.566309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:19.566846 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:19.567064 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:19.567527 (+ 463us) server_negotiation.cc:299] Negotiation successful
0806 17:02:19.567937 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":160,"threads_started":1}
I20260806 17:02:19.569178 10867 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba7035a4ce364e81b797d4307edf3133. Latest consensus state: current_term: 1 leader_uuid: "ba7035a4ce364e81b797d4307edf3133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7035a4ce364e81b797d4307edf3133" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 46561 } } }
I20260806 17:02:19.569309 10867 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:19.569732 10869 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba7035a4ce364e81b797d4307edf3133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7035a4ce364e81b797d4307edf3133" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 46561 } } }
I20260806 17:02:19.569872 10869 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133 [sys.catalog]: This master's current role is: LEADER
W20260806 17:02:19.573805 10883 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:02:19.573930 10883 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:02:19.574007 10873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:19.574919 10873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:02:19.580931 10873 catalog_manager.cc:1347] Generated new cluster ID: 65bf76dfac05421ebad8862dd0740532
I20260806 17:02:19.581027 10873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:19.589181 10873 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:02:19.590576 10873 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:19.595988 10873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133: Generated new TSK 0
I20260806 17:02:19.596801 10873 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
I20260806 17:02:19.633375 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:46561
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:19.759567 10889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:19.759868 10889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:19.759915 10889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:19.764209 10889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:19.764290 10889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:19.764333 10889 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 17:02:19.764369 10889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:02:19.770335 10889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.111.126:46561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:19.771566 10889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:19.772729 10889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:19.783917 10897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:19.783980 10895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:19.783978 10894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:19.784406 10889 server_base.cc:1034] running on GCE node
I20260806 17:02:19.784893 10889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:19.785605 10889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:19.786830 10889 hybrid_clock.cc:648] HybridClock initialized: now 1786035739786807 us; error 69 us; skew 500 ppm
Aug 06 17:02:19 dist-test-slave-69rg krb5kdc[10769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035739, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:19.790097 10889 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:02:19.791154 10889 webserver.cc:460] Webserver started at http://127.9.111.65:42877/ using document root <none> and password file <none>
I20260806 17:02:19.791815 10889 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:19.791921 10889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:19.792172 10889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:19.794485 10889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "d8891eb062b34a4385f896fef711546f"
format_stamp: "Formatted at 2026-08-06 17:02:19 on dist-test-slave-69rg"
server_key: "84e4b13878d47e4f7aabcad2571ea572"
server_key_iv: "00767eafbb01fbdeb7138bc6cc1aefc2"
server_key_version: "encryptionkey@0"
I20260806 17:02:19.795117 10889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "d8891eb062b34a4385f896fef711546f"
format_stamp: "Formatted at 2026-08-06 17:02:19 on dist-test-slave-69rg"
server_key: "84e4b13878d47e4f7aabcad2571ea572"
server_key_iv: "00767eafbb01fbdeb7138bc6cc1aefc2"
server_key_version: "encryptionkey@0"
I20260806 17:02:19.799276 10889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:02:19.802044 10904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:19.803339 10889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:19.803479 10889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "d8891eb062b34a4385f896fef711546f"
format_stamp: "Formatted at 2026-08-06 17:02:19 on dist-test-slave-69rg"
server_key: "84e4b13878d47e4f7aabcad2571ea572"
server_key_iv: "00767eafbb01fbdeb7138bc6cc1aefc2"
server_key_version: "encryptionkey@0"
I20260806 17:02:19.803609 10889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:19.820834 10889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:19.824680 10889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:19.824928 10889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:19.826710 10889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:19.826814 10889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:19.826879 10889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:19.826932 10889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 17:02:19 dist-test-slave-69rg krb5kdc[10769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035739, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:19.840626 10889 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:38925
I20260806 17:02:19.841055 11022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:38925 every 8 connection(s)
I20260806 17:02:19.841768 10889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:02:19.842180 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 10889
I20260806 17:02:19.842334 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:02:19.842664 9661 external_mini_cluster.cc:1428] Setting key aece9b1252fe54655081e0f87d348f58
I20260806 17:02:19.844350 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:19.827821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:42617 (local address 127.9.111.126:46561)
0806 17:02:19.827970 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:19.827974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:19.829281 (+ 1307us) server_negotiation.cc:407] Connection header received
0806 17:02:19.830352 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:19.830356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:19.830418 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:19.830532 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:19.832049 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:19.832824 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:19.835090 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:19.835574 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:19.835705 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:19.839446 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:19.839449 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:19.839451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:19.839481 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:19.841629 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:19.842439 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:19.842443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:19.842445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:19.842564 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:19.842993 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:19.842997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:19.842999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:19.843314 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:19.843449 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:19.844015 (+ 566us) server_negotiation.cc:299] Negotiation successful
0806 17:02:19.844163 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 17:02:19.845201 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:19.828482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:46561 (local address 127.9.111.65:42617)
0806 17:02:19.829070 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:19.829120 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:19.830115 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:19.830721 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:19.830728 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:19.831142 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:19.831856 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:19.831881 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:19.833017 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:19.833024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:19.834887 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:19.834906 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:19.835762 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:19.835766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:19.835980 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:19.836817 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:19.836834 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:19.839161 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:19.841776 (+ 2615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:19.841786 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:19.841808 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:19.842286 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:19.842692 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:19.842696 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:19.842698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:19.842857 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:19.843445 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:19.843451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:19.843735 (+ 284us) client_negotiation.cc:769] Sending connection context
0806 17:02:19.843946 (+ 211us) client_negotiation.cc:240] Negotiation successful
0806 17:02:19.844322 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":157,"threads_started":1}
I20260806 17:02:19.847031 11023 heartbeater.cc:347] Connected to a master server at 127.9.111.126:46561
I20260806 17:02:19.847213 11023 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:19.847801 11023 heartbeater.cc:511] Master 127.9.111.126:46561 requested a full tablet report, sending...
I20260806 17:02:19.849642 10815 ts_manager.cc:194] Registered new tserver with Master: d8891eb062b34a4385f896fef711546f (127.9.111.65:38925)
I20260806 17:02:19.851248 10815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:42617
WARNING: no policy specified for kudu/127.9.111.66@KRBTEST.COM; defaulting to no policy
I20260806 17:02:19.858489 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:19.849549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:60647 (local address 127.9.111.126:46561)
0806 17:02:19.849699 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:19.849705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:19.850040 (+ 335us) server_negotiation.cc:407] Connection header received
0806 17:02:19.850151 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:19.850154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:19.850201 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:19.850303 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:19.850945 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:19.851736 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:19.853179 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:19.853543 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:19.853666 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:19.855138 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:19.855141 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:19.855143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:19.855168 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:19.856772 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:19.857196 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:19.857199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:19.857200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:19.857280 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:19.857558 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:19.857560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:19.857561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:19.857746 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:19.857887 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:19.858227 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 17:02:19.858367 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 17:02:19.858539 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:19.849446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:46561 (local address 127.9.111.65:60647)
0806 17:02:19.849930 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:19.849948 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:19.850063 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:19.850348 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:19.850354 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:19.850611 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:19.850791 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:19.850801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:19.851891 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:19.851898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:19.853019 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:19.853033 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:19.853706 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:19.853712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:19.853857 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:19.854458 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:19.854479 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:19.854974 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:19.856900 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:19.856903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:19.856905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:19.857103 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:19.857392 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:19.857394 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:19.857395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:19.857479 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:19.857922 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:19.857927 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:19.858056 (+ 129us) client_negotiation.cc:769] Sending connection context
0806 17:02:19.858167 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 17:02:19.858306 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
I20260806 17:02:19.878134 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:46561
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:19.993194 11033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:19.993474 11033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:19.993520 11033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:19.997615 11033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:19.997689 11033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:19.997730 11033 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 17:02:19.997805 11033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:02:20.003657 11033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.111.126:46561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:20.004910 11033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:20.006125 11033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:20.018193 11038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:20.018265 11033 server_base.cc:1034] running on GCE node
W20260806 17:02:20.018214 11041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:20.018208 11039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:20.018889 11033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:20.019779 11033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:20.020952 11033 hybrid_clock.cc:648] HybridClock initialized: now 1786035740020904 us; error 61 us; skew 500 ppm
Aug 06 17:02:20 dist-test-slave-69rg krb5kdc[10769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035740, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:20.024415 11033 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:02:20.025513 11033 webserver.cc:460] Webserver started at http://127.9.111.66:39341/ using document root <none> and password file <none>
I20260806 17:02:20.026264 11033 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:20.026360 11033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:20.026569 11033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:20.028892 11033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "3d6869aae8d24fc39e347fa9c9074e9d"
format_stamp: "Formatted at 2026-08-06 17:02:20 on dist-test-slave-69rg"
server_key: "09fd86d093e1e395599f88efe921e24f"
server_key_iv: "92c1f3401908dbcc22d352e84ce7456e"
server_key_version: "encryptionkey@0"
I20260806 17:02:20.029502 11033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "3d6869aae8d24fc39e347fa9c9074e9d"
format_stamp: "Formatted at 2026-08-06 17:02:20 on dist-test-slave-69rg"
server_key: "09fd86d093e1e395599f88efe921e24f"
server_key_iv: "92c1f3401908dbcc22d352e84ce7456e"
server_key_version: "encryptionkey@0"
I20260806 17:02:20.033900 11033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260806 17:02:20.036723 11048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:20.038225 11033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:20.038336 11033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "3d6869aae8d24fc39e347fa9c9074e9d"
format_stamp: "Formatted at 2026-08-06 17:02:20 on dist-test-slave-69rg"
server_key: "09fd86d093e1e395599f88efe921e24f"
server_key_iv: "92c1f3401908dbcc22d352e84ce7456e"
server_key_version: "encryptionkey@0"
I20260806 17:02:20.038478 11033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:20.057964 11033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:20.062167 11033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:20.062413 11033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:20.064404 11033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:20.064467 11033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:20.064523 11033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:20.064555 11033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:20.076661 11033 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:39731
I20260806 17:02:20.076758 11167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:39731 every 8 connection(s)
Aug 06 17:02:20 dist-test-slave-69rg krb5kdc[10769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035740, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:20.077706 11033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:02:20.082190 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.066269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:45031 (local address 127.9.111.126:46561)
0806 17:02:20.066467 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:20.066473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:20.067086 (+ 613us) server_negotiation.cc:407] Connection header received
0806 17:02:20.068931 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:20.068935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:20.069003 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:20.069050 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:20.071290 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.072246 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.074344 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.074779 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.074917 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.078157 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:20.078163 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:20.078169 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:20.078210 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:20.079914 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:20.080443 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:20.080446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:20.080447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:20.080538 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:20.080908 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:20.080912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:20.080913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:20.081140 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:20.081249 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:20.081795 (+ 546us) server_negotiation.cc:299] Negotiation successful
0806 17:02:20.082020 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 17:02:20.083004 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.066153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:46561 (local address 127.9.111.66:45031)
0806 17:02:20.066923 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:20.066978 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:20.068621 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:20.069625 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:20.069633 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:20.070225 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:20.071126 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:20.071154 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.072406 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.072413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:20.074100 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:20.074164 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.075007 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.075014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:20.075287 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.076044 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:20.076061 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:20.077901 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:20.080036 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:20.080041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:20.080054 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:20.080330 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:20.080623 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:20.080625 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:20.080626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:20.080767 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:20.081243 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:20.081248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:20.081500 (+ 252us) client_negotiation.cc:769] Sending connection context
0806 17:02:20.081661 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 17:02:20.082076 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":124,"threads_started":1}
I20260806 17:02:20.084411 11168 heartbeater.cc:347] Connected to a master server at 127.9.111.126:46561
I20260806 17:02:20.084584 11168 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:20.085228 11168 heartbeater.cc:511] Master 127.9.111.126:46561 requested a full tablet report, sending...
I20260806 17:02:20.086447 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 11033
I20260806 17:02:20.086531 10813 ts_manager.cc:194] Registered new tserver with Master: 3d6869aae8d24fc39e347fa9c9074e9d (127.9.111.66:39731)
I20260806 17:02:20.086584 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:02:20.086906 9661 external_mini_cluster.cc:1428] Setting key 23d7acfab9cbc9bf73b5a2c5c30bc865
I20260806 17:02:20.087388 10813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:45031
I20260806 17:02:20.095654 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.087089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:46561 (local address 127.9.111.66:51033)
0806 17:02:20.087459 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:20.087479 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:20.087585 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:20.087924 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:20.087928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:20.088193 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:20.088368 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:20.088378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.089345 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.089348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:20.090076 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:20.090086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.090606 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.090610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:20.090715 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.091378 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:20.091391 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:20.091768 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:20.093842 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:20.093849 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:20.093854 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:20.094086 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:20.094409 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:20.094414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:20.094416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:20.094515 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:20.094989 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:20.094992 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:20.095103 (+ 111us) client_negotiation.cc:769] Sending connection context
0806 17:02:20.095223 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 17:02:20.095393 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":3,"thread_start_us":130,"threads_started":1}
I20260806 17:02:20.095937 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.087502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:51033 (local address 127.9.111.126:46561)
0806 17:02:20.087643 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:20.087647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:20.087679 (+ 32us) server_negotiation.cc:407] Connection header received
0806 17:02:20.087725 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:20.087732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:20.087785 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:20.087875 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:20.088505 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.089194 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.090225 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.090480 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.090605 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.091927 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:20.091931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:20.091936 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:20.091972 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:20.093690 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:20.094198 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:20.094204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:20.094210 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:20.094307 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:20.094624 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:20.094628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:20.094630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:20.094881 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:20.095001 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:20.095296 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 17:02:20.095588 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
WARNING: no policy specified for kudu/127.9.111.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
I20260806 17:02:20.121726 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:46561
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:20.252597 11177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:20.252988 11177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:20.253082 11177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:20.257553 11177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:20.257685 11177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:20.257777 11177 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 17:02:20.257879 11177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:02:20.264022 11177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.111.126:46561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:20.265321 11177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:20.266506 11177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:20.278155 11185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:20.278234 11183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:20.278388 11177 server_base.cc:1034] running on GCE node
W20260806 17:02:20.278273 11182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:20.278977 11177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:20.279789 11177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:20.281050 11177 hybrid_clock.cc:648] HybridClock initialized: now 1786035740281029 us; error 37 us; skew 500 ppm
Aug 06 17:02:20 dist-test-slave-69rg krb5kdc[10769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035740, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:20.284276 11177 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:02:20.285418 11177 webserver.cc:460] Webserver started at http://127.9.111.67:39101/ using document root <none> and password file <none>
I20260806 17:02:20.286106 11177 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:20.286208 11177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:20.286461 11177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:20.288714 11177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "46620f3939c348b781ad69ce3e9e2b6a"
format_stamp: "Formatted at 2026-08-06 17:02:20 on dist-test-slave-69rg"
server_key: "9ceb61a4de16823adc61f0a670fb6698"
server_key_iv: "6d4231c9858d175e77f6b036807c6d9d"
server_key_version: "encryptionkey@0"
I20260806 17:02:20.289455 11177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "46620f3939c348b781ad69ce3e9e2b6a"
format_stamp: "Formatted at 2026-08-06 17:02:20 on dist-test-slave-69rg"
server_key: "9ceb61a4de16823adc61f0a670fb6698"
server_key_iv: "6d4231c9858d175e77f6b036807c6d9d"
server_key_version: "encryptionkey@0"
I20260806 17:02:20.293788 11177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 17:02:20.297078 11192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:20.299113 11177 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 17:02:20.299373 11177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "46620f3939c348b781ad69ce3e9e2b6a"
format_stamp: "Formatted at 2026-08-06 17:02:20 on dist-test-slave-69rg"
server_key: "9ceb61a4de16823adc61f0a670fb6698"
server_key_iv: "6d4231c9858d175e77f6b036807c6d9d"
server_key_version: "encryptionkey@0"
I20260806 17:02:20.299580 11177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:20.316658 11177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:20.320528 11177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:20.320784 11177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:20.322443 11177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:20.322522 11177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:20.322592 11177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:20.322642 11177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 17:02:20 dist-test-slave-69rg krb5kdc[10769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035740, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:20.334605 11177 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:36415
I20260806 17:02:20.334659 11314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:36415 every 8 connection(s)
I20260806 17:02:20.335827 11177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:02:20.337268 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.323462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:41635 (local address 127.9.111.126:46561)
0806 17:02:20.323604 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:20.323608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:20.324738 (+ 1130us) server_negotiation.cc:407] Connection header received
0806 17:02:20.325928 (+ 1190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:20.325931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:20.325999 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:20.326092 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:20.327710 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.328319 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.329793 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.330159 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.330299 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.333196 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:20.333200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:20.333202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:20.333242 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:20.335021 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:20.335540 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:20.335544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:20.335545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:20.335639 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:20.336043 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:20.336047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:20.336049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:20.336282 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:20.336394 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:20.336980 (+ 586us) server_negotiation.cc:299] Negotiation successful
0806 17:02:20.337093 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 17:02:20.338156 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.324049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:46561 (local address 127.9.111.67:41635)
0806 17:02:20.324575 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:20.324620 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:20.325639 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:20.326331 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:20.326338 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:20.326857 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:20.327538 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:20.327555 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.328468 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.328471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:20.329626 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:20.329639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.330371 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.330375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:20.330600 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.331381 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:20.331399 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:20.332997 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:20.335134 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:20.335139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:20.335151 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:20.335428 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:20.335733 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:20.335735 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:20.335736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:20.335932 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:20.336447 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:20.336453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:20.336747 (+ 294us) client_negotiation.cc:769] Sending connection context
0806 17:02:20.336912 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 17:02:20.337166 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":134,"threads_started":1}
I20260806 17:02:20.339432 11315 heartbeater.cc:347] Connected to a master server at 127.9.111.126:46561
I20260806 17:02:20.339570 11315 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:20.340166 11315 heartbeater.cc:511] Master 127.9.111.126:46561 requested a full tablet report, sending...
I20260806 17:02:20.340344 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 11177
I20260806 17:02:20.340476 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:02:20.340785 9661 external_mini_cluster.cc:1428] Setting key b6c14b8ef43ca810f64bda8c5ad14cb2
I20260806 17:02:20.341362 10815 ts_manager.cc:194] Registered new tserver with Master: 46620f3939c348b781ad69ce3e9e2b6a (127.9.111.67:36415)
I20260806 17:02:20.342327 10815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:41635
I20260806 17:02:20.343012 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 17:02:20.352694 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.342025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:48829 (local address 127.9.111.126:46561)
0806 17:02:20.342152 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:20.342159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:20.344501 (+ 2342us) server_negotiation.cc:407] Connection header received
0806 17:02:20.344578 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:20.344584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:20.344652 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:20.344734 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:20.345221 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.346000 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.346814 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.347199 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.347317 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.348892 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:20.348899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:20.348903 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:20.348934 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:20.350457 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:20.350971 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:20.350974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:20.350975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:20.351054 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:20.351477 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:20.351482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:20.351487 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:20.351732 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:20.351841 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:20.352322 (+ 481us) server_negotiation.cc:299] Negotiation successful
0806 17:02:20.352467 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 17:02:20.352959 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.342027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:46561 (local address 127.9.111.67:48829)
0806 17:02:20.344400 (+ 2373us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:20.344414 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:20.344500 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:20.344787 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:20.344789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:20.344978 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:20.345077 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:20.345083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.346123 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.346126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:20.346694 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:20.346702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.347316 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.347319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:20.347400 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.348134 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:20.348147 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:20.348718 (+ 571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:20.350573 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:20.350579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:20.350584 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:20.350867 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:20.351197 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:20.351201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:20.351203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:20.351355 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:20.351928 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:20.351932 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:20.352125 (+ 193us) client_negotiation.cc:769] Sending connection context
0806 17:02:20.352271 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 17:02:20.352522 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2296,"thread_start_us":126,"threads_started":1}
Aug 06 17:02:20 dist-test-slave-69rg krb5kdc[10769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035740, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 17:02:20 dist-test-slave-69rg krb5kdc[10769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035740, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:20.375159 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.366060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.9.111.126:46561)
0806 17:02:20.366234 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:20.366238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:20.366253 (+ 15us) server_negotiation.cc:407] Connection header received
0806 17:02:20.366299 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:20.366301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:20.366342 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:20.366465 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:20.367251 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.367798 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.368716 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.368950 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.369065 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.371853 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:20.371857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:20.371859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:20.371888 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:20.373285 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:20.373763 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:20.373790 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:20.373791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:20.373957 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:20.374247 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:20.374251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:20.374253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:20.374455 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:20.374601 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:20.374857 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 17:02:20.374983 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 17:02:20.378448 10813 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:02:20.381580 10813 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:02:20.399308 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.391631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:39731 (local address 127.0.0.1:38600)
0806 17:02:20.392208 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:20.392260 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:20.392591 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:20.394021 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:20.394027 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:20.394051 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:20.394245 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:20.394269 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.395801 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.395804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:20.397601 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:20.397612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.398642 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.398646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:20.398777 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.398825 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 17:02:20.398971 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 17:02:20.399116 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":324,"thread_start_us":238,"threads_started":1}
I20260806 17:02:20.399638 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.393053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57430 (local address 127.9.111.67:36415)
0806 17:02:20.393396 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:20.393408 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:20.393743 (+ 335us) server_negotiation.cc:407] Connection header received
0806 17:02:20.393964 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:20.393971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:20.394171 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:20.394522 (+ 351us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:20.394706 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.395577 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.397757 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.398625 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.398734 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.398894 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:20.399374 (+ 480us) server_negotiation.cc:299] Negotiation successful
0806 17:02:20.399485 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":122,"threads_started":1}
I20260806 17:02:20.400009 11336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.392775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46374 (local address 127.9.111.65:38925)
0806 17:02:20.393363 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:20.393370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:20.394313 (+ 943us) server_negotiation.cc:407] Connection header received
0806 17:02:20.394460 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:20.394465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:20.394644 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:20.395019 (+ 375us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:20.395593 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.396791 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.398042 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.399013 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.399110 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.399253 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:20.399772 (+ 519us) server_negotiation.cc:299] Negotiation successful
0806 17:02:20.399887 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":401,"threads_started":1}
I20260806 17:02:20.400259 11333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.391841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38600 (local address 127.9.111.66:39731)
0806 17:02:20.393433 (+ 1592us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:20.393446 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:20.393468 (+ 22us) server_negotiation.cc:407] Connection header received
0806 17:02:20.393536 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:20.393540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:20.393709 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:20.394024 (+ 315us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:20.394435 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.395663 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.397743 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.398506 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.398638 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.398830 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:20.399993 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0806 17:02:20.400079 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":111,"threads_started":1}
I20260806 17:02:20.400286 11335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.392614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:38925 (local address 127.0.0.1:46374)
0806 17:02:20.394014 (+ 1400us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:20.394029 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:20.394313 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:20.395198 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:20.395200 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:20.395218 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:20.395317 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:20.395323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.396909 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.396914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:20.397910 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:20.397921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.399311 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.399317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:20.399413 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.399435 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 17:02:20.399504 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 17:02:20.399571 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1334,"mutex_wait_us":93,"thread_start_us":99,"threads_started":1}
I20260806 17:02:20.400579 11334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.391648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:36415 (local address 127.0.0.1:57430)
0806 17:02:20.393475 (+ 1827us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:20.393489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:20.393772 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:20.394488 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:20.394491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:20.394510 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:20.394600 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:20.394618 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.395713 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.395716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:20.397598 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:20.397610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.398965 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.398969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:20.399070 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.399102 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 17:02:20.399180 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 17:02:20.399254 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1762,"mutex_wait_us":152,"thread_start_us":350,"threads_started":1}
I20260806 17:02:20.403218 11244 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0ee9ab97fb4ab4a3be276c68834ea4 (DEFAULT_TABLE table=test-table [id=238f99b2898a40e7ad37f56f4a2f13cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:20.403227 10956 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0ee9ab97fb4ab4a3be276c68834ea4 (DEFAULT_TABLE table=test-table [id=238f99b2898a40e7ad37f56f4a2f13cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:20.404418 10956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f0ee9ab97fb4ab4a3be276c68834ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:20.404441 11244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f0ee9ab97fb4ab4a3be276c68834ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:20.409648 11094 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0ee9ab97fb4ab4a3be276c68834ea4 (DEFAULT_TABLE table=test-table [id=238f99b2898a40e7ad37f56f4a2f13cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:20.410795 11094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f0ee9ab97fb4ab4a3be276c68834ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:20.413280 11338 tablet_bootstrap.cc:492] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a: Bootstrap starting.
I20260806 17:02:20.416615 11338 tablet_bootstrap.cc:654] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:20.417584 11338 log.cc:824] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:20.418267 11339 tablet_bootstrap.cc:492] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f: Bootstrap starting.
I20260806 17:02:20.419272 11341 tablet_bootstrap.cc:492] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d: Bootstrap starting.
I20260806 17:02:20.420377 11338 tablet_bootstrap.cc:492] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a: No bootstrap required, opened a new log
I20260806 17:02:20.420601 11338 ts_tablet_manager.cc:1397] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 17:02:20.420868 11339 tablet_bootstrap.cc:654] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:20.421555 11341 tablet_bootstrap.cc:654] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:20.421877 11339 log.cc:824] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:20.422614 11341 log.cc:824] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:20.424162 11339 tablet_bootstrap.cc:492] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f: No bootstrap required, opened a new log
I20260806 17:02:20.424397 11339 ts_tablet_manager.cc:1397] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 17:02:20.424494 11338 raft_consensus.cc:348] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } } peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } }
I20260806 17:02:20.424773 11338 raft_consensus.cc:374] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:20.424849 11338 raft_consensus.cc:729] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46620f3939c348b781ad69ce3e9e2b6a, State: Initialized, Role: FOLLOWER
I20260806 17:02:20.424871 11341 tablet_bootstrap.cc:492] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d: No bootstrap required, opened a new log
I20260806 17:02:20.425041 11341 ts_tablet_manager.cc:1397] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 17:02:20.425314 11338 consensus_queue.cc:260] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } } peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } }
I20260806 17:02:20.426038 11315 heartbeater.cc:503] Master 127.9.111.126:46561 was elected leader, sending a full tablet report...
I20260806 17:02:20.426429 11338 ts_tablet_manager.cc:1428] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260806 17:02:20.429347 11339 raft_consensus.cc:348] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } } peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } }
I20260806 17:02:20.429347 11341 raft_consensus.cc:348] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } } peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } }
I20260806 17:02:20.429658 11341 raft_consensus.cc:374] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:20.429658 11339 raft_consensus.cc:374] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:20.429733 11339 raft_consensus.cc:729] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8891eb062b34a4385f896fef711546f, State: Initialized, Role: FOLLOWER
I20260806 17:02:20.429755 11341 raft_consensus.cc:729] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6869aae8d24fc39e347fa9c9074e9d, State: Initialized, Role: FOLLOWER
I20260806 17:02:20.430297 11339 consensus_queue.cc:260] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } } peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } }
I20260806 17:02:20.430296 11341 consensus_queue.cc:260] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } } peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } }
I20260806 17:02:20.431437 11339 ts_tablet_manager.cc:1428] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 17:02:20.431437 11341 ts_tablet_manager.cc:1428] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 17:02:20.431777 11023 heartbeater.cc:503] Master 127.9.111.126:46561 was elected leader, sending a full tablet report...
I20260806 17:02:20.432785 11168 heartbeater.cc:503] Master 127.9.111.126:46561 was elected leader, sending a full tablet report...
W20260806 17:02:20.579043 11169 tablet.cc:2369] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 17:02:20.586891 11316 tablet.cc:2369] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 17:02:20.594060 11024 tablet.cc:2369] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:02:20.781847 11345 raft_consensus.cc:482] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:20.782104 11345 raft_consensus.cc:504] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } } peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } }
I20260806 17:02:20.783396 11345 leader_election.cc:288] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d6869aae8d24fc39e347fa9c9074e9d (127.9.111.66:39731), 46620f3939c348b781ad69ce3e9e2b6a (127.9.111.67:36415)
I20260806 17:02:20.789474 11333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.783770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:34591 (local address 127.9.111.66:39731)
0806 17:02:20.783939 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:20.783945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:20.784311 (+ 366us) server_negotiation.cc:407] Connection header received
0806 17:02:20.784433 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:20.784436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:20.784489 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:20.784583 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:20.785203 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.786046 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.788088 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.788751 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.788894 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.788952 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:20.789256 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 17:02:20.789335 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 17:02:20.789577 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.783632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:39731 (local address 127.9.111.65:34591)
0806 17:02:20.784149 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:20.784165 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:20.784317 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:20.784685 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:20.784689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:20.784722 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:20.784981 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:20.784990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.786216 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.786220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:20.787901 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:20.787914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.788907 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.788911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:20.789000 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.789026 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 17:02:20.789205 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 17:02:20.789302 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":230,"spinlock_wait_cycles":15616,"thread_start_us":201,"threads_started":1}
I20260806 17:02:20.789805 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.783946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:46297 (local address 127.9.111.67:36415)
0806 17:02:20.784080 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:20.784083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:20.784311 (+ 228us) server_negotiation.cc:407] Connection header received
0806 17:02:20.784432 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:20.784435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:20.784489 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:20.784583 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:20.785203 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.786067 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.788395 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.789024 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.789128 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.789198 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:20.789532 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 17:02:20.789617 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 17:02:20.789891 11347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.783748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:36415 (local address 127.9.111.65:46297)
0806 17:02:20.784187 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:20.784202 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:20.784317 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:20.784685 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:20.784689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:20.784722 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:20.784981 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:20.784990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.786226 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.786229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:20.787901 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:20.787914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.789217 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.789221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:20.789355 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.789391 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 17:02:20.789474 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 17:02:20.789567 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":224,"spinlock_wait_cycles":28032,"thread_start_us":127,"threads_started":1}
I20260806 17:02:20.790639 11118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0ee9ab97fb4ab4a3be276c68834ea4" candidate_uuid: "d8891eb062b34a4385f896fef711546f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" is_pre_election: true
I20260806 17:02:20.790647 11264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0ee9ab97fb4ab4a3be276c68834ea4" candidate_uuid: "d8891eb062b34a4385f896fef711546f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46620f3939c348b781ad69ce3e9e2b6a" is_pre_election: true
I20260806 17:02:20.790971 11118 raft_consensus.cc:2454] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8891eb062b34a4385f896fef711546f in term 0.
I20260806 17:02:20.790977 11264 raft_consensus.cc:2454] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8891eb062b34a4385f896fef711546f in term 0.
I20260806 17:02:20.791510 10906 leader_election.cc:302] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d6869aae8d24fc39e347fa9c9074e9d, d8891eb062b34a4385f896fef711546f; no voters:
I20260806 17:02:20.791843 11345 raft_consensus.cc:2781] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:02:20.791934 11345 raft_consensus.cc:482] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:02:20.791989 11345 raft_consensus.cc:3037] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:20.793133 11345 raft_consensus.cc:504] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } } peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } }
I20260806 17:02:20.793571 11345 leader_election.cc:288] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [CANDIDATE]: Term 1 election: Requested vote from peers 3d6869aae8d24fc39e347fa9c9074e9d (127.9.111.66:39731), 46620f3939c348b781ad69ce3e9e2b6a (127.9.111.67:36415)
I20260806 17:02:20.794034 11264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0ee9ab97fb4ab4a3be276c68834ea4" candidate_uuid: "d8891eb062b34a4385f896fef711546f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46620f3939c348b781ad69ce3e9e2b6a"
I20260806 17:02:20.794063 11118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0ee9ab97fb4ab4a3be276c68834ea4" candidate_uuid: "d8891eb062b34a4385f896fef711546f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d6869aae8d24fc39e347fa9c9074e9d"
I20260806 17:02:20.794193 11264 raft_consensus.cc:3037] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:20.794219 11118 raft_consensus.cc:3037] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:20.795393 11118 raft_consensus.cc:2454] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8891eb062b34a4385f896fef711546f in term 1.
I20260806 17:02:20.795796 10906 leader_election.cc:302] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d6869aae8d24fc39e347fa9c9074e9d, d8891eb062b34a4385f896fef711546f; no voters:
I20260806 17:02:20.795840 11264 raft_consensus.cc:2454] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8891eb062b34a4385f896fef711546f in term 1.
I20260806 17:02:20.796028 11345 raft_consensus.cc:2781] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:20.796332 11345 raft_consensus.cc:686] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [term 1 LEADER]: Becoming Leader. State: Replica: d8891eb062b34a4385f896fef711546f, State: Running, Role: LEADER
I20260806 17:02:20.796779 11345 consensus_queue.cc:237] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } } peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } }
I20260806 17:02:20.800632 10813 catalog_manager.cc:5495] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f reported cstate change: term changed from 0 to 1, leader changed from <none> to d8891eb062b34a4385f896fef711546f (127.9.111.65). New cstate: current_term: 1 leader_uuid: "d8891eb062b34a4385f896fef711546f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } health_report { overall_health: HEALTHY } } }
I20260806 17:02:20.861630 11336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:20.857588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46384 (local address 127.9.111.65:38925)
0806 17:02:20.857752 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:20.857759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:20.857779 (+ 20us) server_negotiation.cc:407] Connection header received
0806 17:02:20.857909 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:20.857912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:20.857971 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:20.858085 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:20.858418 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.858965 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.859922 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:20.860146 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:20.860261 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:20.860659 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:20.861125 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:20.861264 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:20.861387 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 17:02:20.861458 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 17:02:20.868315 11118 raft_consensus.cc:1261] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d [term 1 FOLLOWER]: Refusing update from remote peer d8891eb062b34a4385f896fef711546f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:20.868341 11264 raft_consensus.cc:1261] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a [term 1 FOLLOWER]: Refusing update from remote peer d8891eb062b34a4385f896fef711546f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:20.869166 11345 consensus_queue.cc:1035] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [LEADER]: Connected to new peer: Peer: permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:20.869385 11349 consensus_queue.cc:1035] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:20.882802 11354 mvcc.cc:205] Tried to move back new op lower bound from 7315602394588065792 to 7315602394306433024. Current Snapshot: MvccSnapshot[applied={T|T < 7315602394588065792 or (T in {7315602394588065792})}]
I20260806 17:02:20.892333 10813 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:20.892622 10813 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:20.895527 10813 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133: Sending DeleteTablet for 3 replicas of tablet 4f0ee9ab97fb4ab4a3be276c68834ea4
I20260806 17:02:20.896467 11094 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f0ee9ab97fb4ab4a3be276c68834ea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:20 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:38600
I20260806 17:02:20.896518 10956 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f0ee9ab97fb4ab4a3be276c68834ea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:20 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:46374
I20260806 17:02:20.897047 11362 tablet_replica.cc:330] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f: stopping tablet replica
I20260806 17:02:20.897009 11244 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f0ee9ab97fb4ab4a3be276c68834ea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:20 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:57430
I20260806 17:02:20.897243 11363 tablet_replica.cc:330] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d: stopping tablet replica
I20260806 17:02:20.897430 11362 raft_consensus.cc:2229] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:02:20.897567 11363 raft_consensus.cc:2229] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:20.897862 11362 raft_consensus.cc:2258] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:20.898026 11363 raft_consensus.cc:2258] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:20.898567 11364 tablet_replica.cc:330] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a: stopping tablet replica
I20260806 17:02:20.898948 11364 raft_consensus.cc:2229] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:20.899291 11364 raft_consensus.cc:2258] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:20.899160 10813 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:02:20.899914 10813 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:02:20.900136 11362 ts_tablet_manager.cc:1905] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:20.900528 11363 ts_tablet_manager.cc:1905] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:20.902369 11364 ts_tablet_manager.cc:1905] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:20.904032 11362 ts_tablet_manager.cc:1918] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:02:20.904206 11362 log.cc:1196] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-0/wal/wals/4f0ee9ab97fb4ab4a3be276c68834ea4
I20260806 17:02:20.904659 11362 ts_tablet_manager.cc:1939] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P d8891eb062b34a4385f896fef711546f: Deleting consensus metadata
I20260806 17:02:20.905764 11364 ts_tablet_manager.cc:1918] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:02:20.905984 11364 log.cc:1196] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/4f0ee9ab97fb4ab4a3be276c68834ea4
I20260806 17:02:20.906371 10801 catalog_manager.cc:4841] TS d8891eb062b34a4385f896fef711546f (127.9.111.65:38925): tablet 4f0ee9ab97fb4ab4a3be276c68834ea4 (table test-table [id=238f99b2898a40e7ad37f56f4a2f13cb]) successfully deleted
I20260806 17:02:20.906430 11364 ts_tablet_manager.cc:1939] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 46620f3939c348b781ad69ce3e9e2b6a: Deleting consensus metadata
I20260806 17:02:20.907234 11244 tablet_service.cc:1467] Processing CreateTablet for tablet ad83ad9ea8164d9d8b09564b3c1685b6 (DEFAULT_TABLE table=test-table [id=fad91b7a82f248d0b5326cb79834537c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:20.907531 11244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad83ad9ea8164d9d8b09564b3c1685b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:20.908347 11094 tablet_service.cc:1467] Processing CreateTablet for tablet ad83ad9ea8164d9d8b09564b3c1685b6 (DEFAULT_TABLE table=test-table [id=fad91b7a82f248d0b5326cb79834537c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:20.908762 11094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad83ad9ea8164d9d8b09564b3c1685b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:20.909965 10801 catalog_manager.cc:4841] TS 46620f3939c348b781ad69ce3e9e2b6a (127.9.111.67:36415): tablet 4f0ee9ab97fb4ab4a3be276c68834ea4 (table test-table [id=238f99b2898a40e7ad37f56f4a2f13cb]) successfully deleted
I20260806 17:02:20.910446 11338 tablet_bootstrap.cc:492] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a: Bootstrap starting.
I20260806 17:02:20.910975 11363 ts_tablet_manager.cc:1918] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:02:20.911077 11363 log.cc:1196] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/4f0ee9ab97fb4ab4a3be276c68834ea4
I20260806 17:02:20.911401 11363 ts_tablet_manager.cc:1939] T 4f0ee9ab97fb4ab4a3be276c68834ea4 P 3d6869aae8d24fc39e347fa9c9074e9d: Deleting consensus metadata
I20260806 17:02:20.911458 11338 tablet_bootstrap.cc:654] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:20.911782 10956 tablet_service.cc:1467] Processing CreateTablet for tablet ad83ad9ea8164d9d8b09564b3c1685b6 (DEFAULT_TABLE table=test-table [id=fad91b7a82f248d0b5326cb79834537c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:20.912072 10956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad83ad9ea8164d9d8b09564b3c1685b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:20.912940 11338 tablet_bootstrap.cc:492] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a: No bootstrap required, opened a new log
I20260806 17:02:20.913014 11338 ts_tablet_manager.cc:1397] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 17:02:20.913486 11338 raft_consensus.cc:348] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } }
I20260806 17:02:20.913645 11338 raft_consensus.cc:374] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:20.913686 11338 raft_consensus.cc:729] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46620f3939c348b781ad69ce3e9e2b6a, State: Initialized, Role: FOLLOWER
I20260806 17:02:20.913803 11341 tablet_bootstrap.cc:492] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d: Bootstrap starting.
I20260806 17:02:20.914094 10799 catalog_manager.cc:4841] TS 3d6869aae8d24fc39e347fa9c9074e9d (127.9.111.66:39731): tablet 4f0ee9ab97fb4ab4a3be276c68834ea4 (table test-table [id=238f99b2898a40e7ad37f56f4a2f13cb]) successfully deleted
I20260806 17:02:20.913836 11338 consensus_queue.cc:260] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } }
I20260806 17:02:20.914636 11338 ts_tablet_manager.cc:1428] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 17:02:20.914696 11339 tablet_bootstrap.cc:492] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f: Bootstrap starting.
I20260806 17:02:20.914808 11341 tablet_bootstrap.cc:654] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:20.915616 11339 tablet_bootstrap.cc:654] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:20.917233 11339 tablet_bootstrap.cc:492] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f: No bootstrap required, opened a new log
I20260806 17:02:20.917325 11341 tablet_bootstrap.cc:492] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d: No bootstrap required, opened a new log
I20260806 17:02:20.917361 11339 ts_tablet_manager.cc:1397] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 17:02:20.917418 11341 ts_tablet_manager.cc:1397] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260806 17:02:20.917773 11339 raft_consensus.cc:348] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } }
I20260806 17:02:20.917883 11341 raft_consensus.cc:348] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } }
I20260806 17:02:20.917975 11339 raft_consensus.cc:374] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:20.917994 11341 raft_consensus.cc:374] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:20.918025 11341 raft_consensus.cc:729] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6869aae8d24fc39e347fa9c9074e9d, State: Initialized, Role: FOLLOWER
I20260806 17:02:20.918021 11339 raft_consensus.cc:729] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8891eb062b34a4385f896fef711546f, State: Initialized, Role: FOLLOWER
I20260806 17:02:20.918183 11341 consensus_queue.cc:260] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } }
I20260806 17:02:20.918185 11339 consensus_queue.cc:260] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } }
I20260806 17:02:20.918445 11341 ts_tablet_manager.cc:1428] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 17:02:20.918473 11339 ts_tablet_manager.cc:1428] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 17:02:20.980541 11345 raft_consensus.cc:482] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:20.980705 11345 raft_consensus.cc:504] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } }
I20260806 17:02:20.981189 11345 leader_election.cc:288] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d6869aae8d24fc39e347fa9c9074e9d (127.9.111.66:39731), 46620f3939c348b781ad69ce3e9e2b6a (127.9.111.67:36415)
I20260806 17:02:20.981647 11264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad83ad9ea8164d9d8b09564b3c1685b6" candidate_uuid: "d8891eb062b34a4385f896fef711546f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46620f3939c348b781ad69ce3e9e2b6a" is_pre_election: true
I20260806 17:02:20.981645 11118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad83ad9ea8164d9d8b09564b3c1685b6" candidate_uuid: "d8891eb062b34a4385f896fef711546f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" is_pre_election: true
I20260806 17:02:20.981804 11264 raft_consensus.cc:2454] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8891eb062b34a4385f896fef711546f in term 0.
I20260806 17:02:20.981837 11118 raft_consensus.cc:2454] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8891eb062b34a4385f896fef711546f in term 0.
I20260806 17:02:20.982146 10906 leader_election.cc:302] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d6869aae8d24fc39e347fa9c9074e9d, d8891eb062b34a4385f896fef711546f; no voters:
I20260806 17:02:20.982345 11345 raft_consensus.cc:2781] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:02:20.982425 11345 raft_consensus.cc:482] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:02:20.982479 11345 raft_consensus.cc:3037] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:20.983435 11345 raft_consensus.cc:504] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } }
I20260806 17:02:20.983788 11345 leader_election.cc:288] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [CANDIDATE]: Term 1 election: Requested vote from peers 3d6869aae8d24fc39e347fa9c9074e9d (127.9.111.66:39731), 46620f3939c348b781ad69ce3e9e2b6a (127.9.111.67:36415)
I20260806 17:02:20.984206 11264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad83ad9ea8164d9d8b09564b3c1685b6" candidate_uuid: "d8891eb062b34a4385f896fef711546f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46620f3939c348b781ad69ce3e9e2b6a"
I20260806 17:02:20.984198 11118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad83ad9ea8164d9d8b09564b3c1685b6" candidate_uuid: "d8891eb062b34a4385f896fef711546f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d6869aae8d24fc39e347fa9c9074e9d"
I20260806 17:02:20.984372 11118 raft_consensus.cc:3037] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:20.984375 11264 raft_consensus.cc:3037] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:20.985404 11264 raft_consensus.cc:2454] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8891eb062b34a4385f896fef711546f in term 1.
I20260806 17:02:20.985610 11118 raft_consensus.cc:2454] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8891eb062b34a4385f896fef711546f in term 1.
I20260806 17:02:20.985697 10908 leader_election.cc:302] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46620f3939c348b781ad69ce3e9e2b6a, d8891eb062b34a4385f896fef711546f; no voters:
I20260806 17:02:20.985917 11345 raft_consensus.cc:2781] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:20.986011 11345 raft_consensus.cc:686] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [term 1 LEADER]: Becoming Leader. State: Replica: d8891eb062b34a4385f896fef711546f, State: Running, Role: LEADER
I20260806 17:02:20.986153 11345 consensus_queue.cc:237] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } }
I20260806 17:02:20.988056 10815 catalog_manager.cc:5495] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f reported cstate change: term changed from 0 to 1, leader changed from <none> to d8891eb062b34a4385f896fef711546f (127.9.111.65). New cstate: current_term: 1 leader_uuid: "d8891eb062b34a4385f896fef711546f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } health_report { overall_health: UNKNOWN } } }
Aug 06 17:02:21 dist-test-slave-69rg krb5kdc[10769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035739, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:21.029500 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:21.020015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41104 (local address 127.9.111.126:46561)
0806 17:02:21.020391 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:21.020400 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:21.020517 (+ 117us) server_negotiation.cc:407] Connection header received
0806 17:02:21.020692 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:21.020705 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:21.020757 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:21.020897 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:21.021668 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.022322 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.023125 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.023357 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.023467 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:21.025944 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:21.025947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:21.025949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:21.025970 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:21.027562 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:21.028070 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:21.028075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:21.028079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:21.028178 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:21.028479 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:21.028484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:21.028486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:21.028683 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:21.028787 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:21.029092 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 17:02:21.029324 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":272,"thread_start_us":164,"threads_started":1}
I20260806 17:02:21.029508 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:21.019968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:46561 (local address 127.0.0.1:41104)
0806 17:02:21.020397 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:21.020415 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:21.020582 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:21.020916 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:21.020919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:21.021422 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:21.021531 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:21.021537 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.022436 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.022440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:21.022997 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:21.023004 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.023474 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.023477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:21.023580 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:21.024238 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:21.024250 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:21.025754 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:21.027677 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:21.027691 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:21.027706 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:21.027953 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:21.028291 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:21.028294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:21.028296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:21.028388 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:21.028801 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:21.028810 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:21.028929 (+ 119us) client_negotiation.cc:769] Sending connection context
0806 17:02:21.029047 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 17:02:21.029266 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":4,"thread_start_us":174,"threads_started":1}
I20260806 17:02:21.035415 10815 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:41104:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 17:02:21.036302 10815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:02:21.041960 10956 tablet_service.cc:1467] Processing CreateTablet for tablet fe869e432a09414da33ff6e84fffc18d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=88a2f597e71846bb8f0be66fc5ffb6d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 17:02:21.042148 11244 tablet_service.cc:1467] Processing CreateTablet for tablet fe869e432a09414da33ff6e84fffc18d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=88a2f597e71846bb8f0be66fc5ffb6d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 17:02:21.042143 11094 tablet_service.cc:1467] Processing CreateTablet for tablet fe869e432a09414da33ff6e84fffc18d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=88a2f597e71846bb8f0be66fc5ffb6d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 17:02:21.042439 10956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe869e432a09414da33ff6e84fffc18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:21.042475 11244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe869e432a09414da33ff6e84fffc18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:21.042526 11094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe869e432a09414da33ff6e84fffc18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:21.045457 11341 tablet_bootstrap.cc:492] T fe869e432a09414da33ff6e84fffc18d P 3d6869aae8d24fc39e347fa9c9074e9d: Bootstrap starting.
I20260806 17:02:21.045646 11338 tablet_bootstrap.cc:492] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a: Bootstrap starting.
I20260806 17:02:21.046474 11338 tablet_bootstrap.cc:654] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:21.046557 11341 tablet_bootstrap.cc:654] T fe869e432a09414da33ff6e84fffc18d P 3d6869aae8d24fc39e347fa9c9074e9d: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:21.047789 11339 tablet_bootstrap.cc:492] T fe869e432a09414da33ff6e84fffc18d P d8891eb062b34a4385f896fef711546f: Bootstrap starting.
I20260806 17:02:21.048379 11338 tablet_bootstrap.cc:492] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a: No bootstrap required, opened a new log
I20260806 17:02:21.048475 11338 ts_tablet_manager.cc:1397] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 17:02:21.048913 11338 raft_consensus.cc:348] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } }
I20260806 17:02:21.049036 11338 raft_consensus.cc:374] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:21.049068 11338 raft_consensus.cc:729] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46620f3939c348b781ad69ce3e9e2b6a, State: Initialized, Role: FOLLOWER
I20260806 17:02:21.049124 11339 tablet_bootstrap.cc:654] T fe869e432a09414da33ff6e84fffc18d P d8891eb062b34a4385f896fef711546f: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:21.049196 11338 consensus_queue.cc:260] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } }
I20260806 17:02:21.049453 11338 ts_tablet_manager.cc:1428] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 17:02:21.050364 11344 tablet_replica.cc:439] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } } }
I20260806 17:02:21.050521 11344 tablet_replica.cc:442] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:21.050519 11339 tablet_bootstrap.cc:492] T fe869e432a09414da33ff6e84fffc18d P d8891eb062b34a4385f896fef711546f: No bootstrap required, opened a new log
I20260806 17:02:21.050602 11339 ts_tablet_manager.cc:1397] T fe869e432a09414da33ff6e84fffc18d P d8891eb062b34a4385f896fef711546f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 17:02:21.051030 11339 raft_consensus.cc:348] T fe869e432a09414da33ff6e84fffc18d P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } }
I20260806 17:02:21.051149 11339 raft_consensus.cc:374] T fe869e432a09414da33ff6e84fffc18d P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:21.051183 11339 raft_consensus.cc:729] T fe869e432a09414da33ff6e84fffc18d P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8891eb062b34a4385f896fef711546f, State: Initialized, Role: FOLLOWER
I20260806 17:02:21.051318 11341 tablet_bootstrap.cc:492] T fe869e432a09414da33ff6e84fffc18d P 3d6869aae8d24fc39e347fa9c9074e9d: No bootstrap required, opened a new log
I20260806 17:02:21.051411 11341 ts_tablet_manager.cc:1397] T fe869e432a09414da33ff6e84fffc18d P 3d6869aae8d24fc39e347fa9c9074e9d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 17:02:21.051414 11339 consensus_queue.cc:260] T fe869e432a09414da33ff6e84fffc18d P d8891eb062b34a4385f896fef711546f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } }
I20260806 17:02:21.051723 11345 tablet_replica.cc:439] T fe869e432a09414da33ff6e84fffc18d P d8891eb062b34a4385f896fef711546f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } } }
I20260806 17:02:21.051842 11345 tablet_replica.cc:442] T fe869e432a09414da33ff6e84fffc18d P d8891eb062b34a4385f896fef711546f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:21.051875 11341 raft_consensus.cc:348] T fe869e432a09414da33ff6e84fffc18d P 3d6869aae8d24fc39e347fa9c9074e9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } }
I20260806 17:02:21.051998 11341 raft_consensus.cc:374] T fe869e432a09414da33ff6e84fffc18d P 3d6869aae8d24fc39e347fa9c9074e9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:21.052035 11341 raft_consensus.cc:729] T fe869e432a09414da33ff6e84fffc18d P 3d6869aae8d24fc39e347fa9c9074e9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6869aae8d24fc39e347fa9c9074e9d, State: Initialized, Role: FOLLOWER
I20260806 17:02:21.052150 11341 consensus_queue.cc:260] T fe869e432a09414da33ff6e84fffc18d P 3d6869aae8d24fc39e347fa9c9074e9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } }
I20260806 17:02:21.052397 11346 tablet_replica.cc:439] T fe869e432a09414da33ff6e84fffc18d P 3d6869aae8d24fc39e347fa9c9074e9d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } } }
I20260806 17:02:21.052554 11346 tablet_replica.cc:442] T fe869e432a09414da33ff6e84fffc18d P 3d6869aae8d24fc39e347fa9c9074e9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:21.053181 11339 ts_tablet_manager.cc:1428] T fe869e432a09414da33ff6e84fffc18d P d8891eb062b34a4385f896fef711546f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260806 17:02:21.052397 11341 ts_tablet_manager.cc:1428] T fe869e432a09414da33ff6e84fffc18d P 3d6869aae8d24fc39e347fa9c9074e9d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 17:02:21.244128 11344 raft_consensus.cc:482] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:21.244321 11344 raft_consensus.cc:504] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } }
I20260806 17:02:21.245383 11344 leader_election.cc:288] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8891eb062b34a4385f896fef711546f (127.9.111.65:38925), 3d6869aae8d24fc39e347fa9c9074e9d (127.9.111.66:39731)
I20260806 17:02:21.250272 11336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:21.245893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:41115 (local address 127.9.111.65:38925)
0806 17:02:21.246025 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:21.246031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:21.246384 (+ 353us) server_negotiation.cc:407] Connection header received
0806 17:02:21.246563 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:21.246566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:21.246620 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:21.246716 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:21.247016 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.247707 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.249180 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.249591 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.249724 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:21.249793 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:21.250052 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 17:02:21.250135 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 17:02:21.250384 11333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:21.245862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:46043 (local address 127.9.111.66:39731)
0806 17:02:21.246031 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:21.246037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:21.246561 (+ 524us) server_negotiation.cc:407] Connection header received
0806 17:02:21.246815 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:21.246819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:21.246875 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:21.246973 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:21.247304 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.247955 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.249155 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.249615 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.249772 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:21.249860 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:21.250176 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 17:02:21.250259 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 17:02:21.250650 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:21.245734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:39731 (local address 127.9.111.67:46043)
0806 17:02:21.246336 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:21.246349 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:21.246607 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:21.246984 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:21.246987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:21.247004 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:21.247176 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:21.247182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.248076 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.248078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:21.249008 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:21.249018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.249783 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.249786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:21.249931 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:21.249948 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 17:02:21.250049 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 17:02:21.250124 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":61,"thread_start_us":140,"threads_started":1}
I20260806 17:02:21.250674 11379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:21.245734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:38925 (local address 127.9.111.67:41115)
0806 17:02:21.246191 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:21.246204 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:21.246446 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:21.246757 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:21.246760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:21.246776 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:21.246898 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:21.246904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.247855 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.247859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:21.249026 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:21.249037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.249735 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.249739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:21.249894 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:21.249919 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 17:02:21.250105 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 17:02:21.250201 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":35,"thread_start_us":314,"threads_started":1}
I20260806 17:02:21.251504 11118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe869e432a09414da33ff6e84fffc18d" candidate_uuid: "46620f3939c348b781ad69ce3e9e2b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" is_pre_election: true
I20260806 17:02:21.251618 10976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe869e432a09414da33ff6e84fffc18d" candidate_uuid: "46620f3939c348b781ad69ce3e9e2b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8891eb062b34a4385f896fef711546f" is_pre_election: true
I20260806 17:02:21.251717 11118 raft_consensus.cc:2454] T fe869e432a09414da33ff6e84fffc18d P 3d6869aae8d24fc39e347fa9c9074e9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46620f3939c348b781ad69ce3e9e2b6a in term 0.
I20260806 17:02:21.251861 10976 raft_consensus.cc:2454] T fe869e432a09414da33ff6e84fffc18d P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46620f3939c348b781ad69ce3e9e2b6a in term 0.
I20260806 17:02:21.252381 11196 leader_election.cc:302] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3d6869aae8d24fc39e347fa9c9074e9d, 46620f3939c348b781ad69ce3e9e2b6a, d8891eb062b34a4385f896fef711546f; no voters:
I20260806 17:02:21.252705 11344 raft_consensus.cc:2781] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:02:21.252774 11344 raft_consensus.cc:482] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:02:21.252813 11344 raft_consensus.cc:3037] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:21.253991 11344 raft_consensus.cc:504] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } }
I20260806 17:02:21.254407 11344 leader_election.cc:288] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a [CANDIDATE]: Term 1 election: Requested vote from peers d8891eb062b34a4385f896fef711546f (127.9.111.65:38925), 3d6869aae8d24fc39e347fa9c9074e9d (127.9.111.66:39731)
I20260806 17:02:21.254829 10976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe869e432a09414da33ff6e84fffc18d" candidate_uuid: "46620f3939c348b781ad69ce3e9e2b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8891eb062b34a4385f896fef711546f"
I20260806 17:02:21.254972 10976 raft_consensus.cc:3037] T fe869e432a09414da33ff6e84fffc18d P d8891eb062b34a4385f896fef711546f [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:21.254926 11118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe869e432a09414da33ff6e84fffc18d" candidate_uuid: "46620f3939c348b781ad69ce3e9e2b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d6869aae8d24fc39e347fa9c9074e9d"
I20260806 17:02:21.255115 11118 raft_consensus.cc:3037] T fe869e432a09414da33ff6e84fffc18d P 3d6869aae8d24fc39e347fa9c9074e9d [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:21.256086 10976 raft_consensus.cc:2454] T fe869e432a09414da33ff6e84fffc18d P d8891eb062b34a4385f896fef711546f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46620f3939c348b781ad69ce3e9e2b6a in term 1.
I20260806 17:02:21.256214 11118 raft_consensus.cc:2454] T fe869e432a09414da33ff6e84fffc18d P 3d6869aae8d24fc39e347fa9c9074e9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46620f3939c348b781ad69ce3e9e2b6a in term 1.
I20260806 17:02:21.256460 11196 leader_election.cc:302] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46620f3939c348b781ad69ce3e9e2b6a, d8891eb062b34a4385f896fef711546f; no voters:
I20260806 17:02:21.256659 11344 raft_consensus.cc:2781] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:21.256887 11344 raft_consensus.cc:686] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a [term 1 LEADER]: Becoming Leader. State: Replica: 46620f3939c348b781ad69ce3e9e2b6a, State: Running, Role: LEADER
I20260806 17:02:21.257345 11344 consensus_queue.cc:237] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } }
I20260806 17:02:21.259047 11381 tablet_replica.cc:439] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a: TxnStatusTablet state changed. Reason: New leader 46620f3939c348b781ad69ce3e9e2b6a. Latest consensus state: current_term: 1 leader_uuid: "46620f3939c348b781ad69ce3e9e2b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } } }
I20260806 17:02:21.259186 11381 tablet_replica.cc:442] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a: This TxnStatusTablet replica's current role is: LEADER
I20260806 17:02:21.259639 11383 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 17:02:21.259835 11383 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 17:02:21.260569 10814 catalog_manager.cc:5495] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46620f3939c348b781ad69ce3e9e2b6a (127.9.111.67). New cstate: current_term: 1 leader_uuid: "46620f3939c348b781ad69ce3e9e2b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } health_report { overall_health: HEALTHY } } }
I20260806 17:02:21.299039 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:21.294484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:36415 (local address 127.0.0.1:57438)
0806 17:02:21.294638 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:21.294665 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:21.294801 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:21.295240 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:21.295242 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:21.295250 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 17:02:21.295373 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:21.295378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.296238 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.296242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:21.296878 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:21.296886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.297400 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.297404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:21.297510 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:21.297565 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 17:02:21.298575 (+ 1010us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 17:02:21.298601 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 17:02:21.298670 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 17:02:21.298901 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260806 17:02:21.299043 11386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:21.294579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57438 (local address 127.9.111.67:36415)
0806 17:02:21.294888 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:21.294893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:21.294910 (+ 17us) server_negotiation.cc:407] Connection header received
0806 17:02:21.294992 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:21.294995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:21.295049 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:21.295221 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:21.295511 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.296090 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.297003 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.297268 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.297397 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:21.297751 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:21.298368 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:21.298641 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:21.298682 (+ 41us) server_negotiation.cc:299] Negotiation successful
0806 17:02:21.298828 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":101,"threads_started":1}
I20260806 17:02:21.302619 11118 raft_consensus.cc:1261] T fe869e432a09414da33ff6e84fffc18d P 3d6869aae8d24fc39e347fa9c9074e9d [term 1 FOLLOWER]: Refusing update from remote peer 46620f3939c348b781ad69ce3e9e2b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:21.302696 10976 raft_consensus.cc:1261] T fe869e432a09414da33ff6e84fffc18d P d8891eb062b34a4385f896fef711546f [term 1 FOLLOWER]: Refusing update from remote peer 46620f3939c348b781ad69ce3e9e2b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:21.303243 11381 consensus_queue.cc:1035] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:21.303458 11344 consensus_queue.cc:1035] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:21.305246 11345 tablet_replica.cc:439] T fe869e432a09414da33ff6e84fffc18d P d8891eb062b34a4385f896fef711546f: TxnStatusTablet state changed. Reason: New leader 46620f3939c348b781ad69ce3e9e2b6a. Latest consensus state: current_term: 1 leader_uuid: "46620f3939c348b781ad69ce3e9e2b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } } }
I20260806 17:02:21.305382 11345 tablet_replica.cc:442] T fe869e432a09414da33ff6e84fffc18d P d8891eb062b34a4385f896fef711546f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:21.305689 11346 tablet_replica.cc:439] T fe869e432a09414da33ff6e84fffc18d P 3d6869aae8d24fc39e347fa9c9074e9d: TxnStatusTablet state changed. Reason: New leader 46620f3939c348b781ad69ce3e9e2b6a. Latest consensus state: current_term: 1 leader_uuid: "46620f3939c348b781ad69ce3e9e2b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } } }
I20260806 17:02:21.306255 11346 tablet_replica.cc:442] T fe869e432a09414da33ff6e84fffc18d P 3d6869aae8d24fc39e347fa9c9074e9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:21.308117 11344 tablet_replica.cc:439] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46620f3939c348b781ad69ce3e9e2b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } } }
I20260806 17:02:21.308282 11344 tablet_replica.cc:442] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a: This TxnStatusTablet replica's current role is: LEADER
I20260806 17:02:21.308713 11346 tablet_replica.cc:439] T fe869e432a09414da33ff6e84fffc18d P 3d6869aae8d24fc39e347fa9c9074e9d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46620f3939c348b781ad69ce3e9e2b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } } }
I20260806 17:02:21.308825 11346 tablet_replica.cc:442] T fe869e432a09414da33ff6e84fffc18d P 3d6869aae8d24fc39e347fa9c9074e9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:21.309939 11353 mvcc.cc:205] Tried to move back new op lower bound from 7315602396369362944 to 7315602396193398784. Current Snapshot: MvccSnapshot[applied={T|T < 7315602396369362944 or (T in {7315602396369362944})}]
I20260806 17:02:21.312527 11344 tablet_replica.cc:439] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46620f3939c348b781ad69ce3e9e2b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } } }
I20260806 17:02:21.312744 11344 tablet_replica.cc:442] T fe869e432a09414da33ff6e84fffc18d P 46620f3939c348b781ad69ce3e9e2b6a: This TxnStatusTablet replica's current role is: LEADER
I20260806 17:02:21.314026 11352 mvcc.cc:205] Tried to move back new op lower bound from 7315602396369362944 to 7315602396193398784. Current Snapshot: MvccSnapshot[applied={T|T < 7315602396369362944}]
I20260806 17:02:21.314230 11345 tablet_replica.cc:439] T fe869e432a09414da33ff6e84fffc18d P d8891eb062b34a4385f896fef711546f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46620f3939c348b781ad69ce3e9e2b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891eb062b34a4385f896fef711546f" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 38925 } } peers { permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 } } peers { permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 } } }
I20260806 17:02:21.314337 11345 tablet_replica.cc:442] T fe869e432a09414da33ff6e84fffc18d P d8891eb062b34a4385f896fef711546f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:21.326535 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:21.322191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:36415 (local address 127.9.111.65:51757)
0806 17:02:21.322589 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:21.322625 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:21.322920 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:21.323224 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:21.323226 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:21.323235 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 17:02:21.323364 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:21.323369 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.324376 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.324383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:21.325167 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:21.325179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.325689 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.325693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:21.325793 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:21.325899 (+ 106us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 17:02:21.326230 (+ 331us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 17:02:21.326241 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 17:02:21.326308 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 17:02:21.326406 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":116,"threads_started":1}
I20260806 17:02:21.326532 11386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:21.322270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:51757 (local address 127.9.111.67:36415)
0806 17:02:21.322439 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:21.322446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:21.322858 (+ 412us) server_negotiation.cc:407] Connection header received
0806 17:02:21.323037 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:21.323040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:21.323120 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:21.323208 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:21.323558 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.324237 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.325307 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.325552 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.325685 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:21.326052 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:21.326142 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:21.326226 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:21.326319 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 17:02:21.326376 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 17:02:21.335773 11264 raft_consensus.cc:1261] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a [term 1 FOLLOWER]: Refusing update from remote peer d8891eb062b34a4385f896fef711546f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:21.335891 11118 raft_consensus.cc:1261] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d [term 1 FOLLOWER]: Refusing update from remote peer d8891eb062b34a4385f896fef711546f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:21.336341 11345 consensus_queue.cc:1035] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d6869aae8d24fc39e347fa9c9074e9d" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 39731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:21.336635 11345 consensus_queue.cc:1035] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [LEADER]: Connected to new peer: Peer: permanent_uuid: "46620f3939c348b781ad69ce3e9e2b6a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:21.371119 11336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:21.367223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:41239 (local address 127.9.111.65:38925)
0806 17:02:21.367330 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:21.367334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:21.367351 (+ 17us) server_negotiation.cc:407] Connection header received
0806 17:02:21.367410 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:21.367412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:21.367452 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:21.367516 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:21.367874 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.368485 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.369876 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.370211 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.370327 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:21.370622 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:21.370696 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:21.370752 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:21.370891 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 17:02:21.370949 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 17:02:21.371122 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:21.366807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:38925 (local address 127.9.111.67:41239)
0806 17:02:21.367228 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:21.367245 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:21.367339 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:21.367549 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:21.367552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:21.367560 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 17:02:21.367744 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:21.367751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.368587 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.368591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:21.369718 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:21.369730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.370339 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.370343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:21.370438 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:21.370491 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 17:02:21.370805 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 17:02:21.370812 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 17:02:21.370883 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 17:02:21.370971 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":145,"threads_started":1}
I20260806 17:02:21.402644 10814 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:21.402846 10814 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:21.405519 10814 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba7035a4ce364e81b797d4307edf3133: Sending DeleteTablet for 3 replicas of tablet ad83ad9ea8164d9d8b09564b3c1685b6
I20260806 17:02:21.406358 10956 tablet_service.cc:1514] Processing DeleteTablet for tablet ad83ad9ea8164d9d8b09564b3c1685b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:21 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:46374
I20260806 17:02:21.406752 11094 tablet_service.cc:1514] Processing DeleteTablet for tablet ad83ad9ea8164d9d8b09564b3c1685b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:21 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:38600
I20260806 17:02:21.406873 11244 tablet_service.cc:1514] Processing DeleteTablet for tablet ad83ad9ea8164d9d8b09564b3c1685b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:21 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:57430
I20260806 17:02:21.406951 11363 tablet_replica.cc:330] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d: stopping tablet replica
I20260806 17:02:21.407084 11363 raft_consensus.cc:2229] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:21.407158 11399 tablet_replica.cc:330] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f: stopping tablet replica
I20260806 17:02:21.407235 11363 raft_consensus.cc:2258] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:21.407274 11399 raft_consensus.cc:2229] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:02:21.407495 11399 raft_consensus.cc:2258] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:21.407730 11364 tablet_replica.cc:330] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a: stopping tablet replica
I20260806 17:02:21.407869 11364 raft_consensus.cc:2229] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:21.407994 11364 raft_consensus.cc:2258] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:21.408459 11363 ts_tablet_manager.cc:1905] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:21.408864 11364 ts_tablet_manager.cc:1905] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:21.410106 11399 ts_tablet_manager.cc:1905] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:21.412067 11364 ts_tablet_manager.cc:1918] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 17:02:21.412148 11364 log.cc:1196] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/ad83ad9ea8164d9d8b09564b3c1685b6
I20260806 17:02:21.412431 11364 ts_tablet_manager.cc:1939] T ad83ad9ea8164d9d8b09564b3c1685b6 P 46620f3939c348b781ad69ce3e9e2b6a: Deleting consensus metadata
I20260806 17:02:21.412603 11363 ts_tablet_manager.cc:1918] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 17:02:21.412688 11363 log.cc:1196] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/ad83ad9ea8164d9d8b09564b3c1685b6
I20260806 17:02:21.412985 11363 ts_tablet_manager.cc:1939] T ad83ad9ea8164d9d8b09564b3c1685b6 P 3d6869aae8d24fc39e347fa9c9074e9d: Deleting consensus metadata
I20260806 17:02:21.413331 10801 catalog_manager.cc:4841] TS 46620f3939c348b781ad69ce3e9e2b6a (127.9.111.67:36415): tablet ad83ad9ea8164d9d8b09564b3c1685b6 (table test-table [id=fad91b7a82f248d0b5326cb79834537c]) successfully deleted
I20260806 17:02:21.413375 11399 ts_tablet_manager.cc:1918] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 17:02:21.413452 11399 log.cc:1196] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786035732873553-9661-0/minicluster-data/ts-0/wal/wals/ad83ad9ea8164d9d8b09564b3c1685b6
I20260806 17:02:21.413767 11399 ts_tablet_manager.cc:1939] T ad83ad9ea8164d9d8b09564b3c1685b6 P d8891eb062b34a4385f896fef711546f: Deleting consensus metadata
I20260806 17:02:21.414021 10799 catalog_manager.cc:4841] TS 3d6869aae8d24fc39e347fa9c9074e9d (127.9.111.66:39731): tablet ad83ad9ea8164d9d8b09564b3c1685b6 (table test-table [id=fad91b7a82f248d0b5326cb79834537c]) successfully deleted
I20260806 17:02:21.414507 10801 catalog_manager.cc:4841] TS d8891eb062b34a4385f896fef711546f (127.9.111.65:38925): tablet ad83ad9ea8164d9d8b09564b3c1685b6 (table test-table [id=fad91b7a82f248d0b5326cb79834537c]) successfully deleted
Aug 06 17:02:21 dist-test-slave-69rg krb5kdc[10769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035740, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.111.65@KRBTEST.COM
I20260806 17:02:21.427680 11336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:21.408249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46390 (local address 127.9.111.65:38925)
0806 17:02:21.409091 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:21.409096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:21.414903 (+ 5807us) server_negotiation.cc:407] Connection header received
0806 17:02:21.415133 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:21.415136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:21.415186 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:21.415262 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:21.416345 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.416984 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.418307 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.418601 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.418704 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:21.422430 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:21.422447 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:21.422464 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:21.422501 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:21.424868 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:21.425761 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:21.425769 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:21.425770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:21.425982 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:21.426503 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:21.426507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:21.426509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:21.426856 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:21.427175 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:21.427343 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 17:02:21.427493 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750}
W20260806 17:02:21.428336 11017 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46390
I20260806 17:02:21.440086 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:21.430897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41110 (local address 127.9.111.126:46561)
0806 17:02:21.431064 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:21.431070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:21.431180 (+ 110us) server_negotiation.cc:407] Connection header received
0806 17:02:21.431381 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:21.431384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:21.431435 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:21.431512 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:21.432385 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.432979 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.434208 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:21.434480 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:21.434570 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:21.436275 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:21.436278 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:21.436280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:21.436311 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:21.438143 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:21.438669 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:21.438671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:21.438673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:21.438763 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:21.439099 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:21.439106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:21.439108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:21.439371 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:21.439483 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:21.439746 (+ 263us) server_negotiation.cc:299] Negotiation successful
0806 17:02:21.439870 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260806 17:02:21.440680 10814 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41110
I20260806 17:02:21.442205 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 10889
I20260806 17:02:21.452911 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 11033
I20260806 17:02:21.460489 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 11177
I20260806 17:02:21.468873 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 10782
2026-08-06T17:02:21Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3343 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:02:21 dist-test-slave-69rg krb5kdc[11406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:02:21 dist-test-slave-69rg krb5kdc[11406](info): set up 2 sockets
Aug 06 17:02:21 dist-test-slave-69rg krb5kdc[11406](info): commencing operation
krb5kdc: starting...
W20260806 17:02:22.554550 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:02:22 dist-test-slave-69rg krb5kdc[11406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:02:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:02:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:02:22.689882 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:43945
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:43945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:22.806406 11420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:22.806734 11420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:22.806810 11420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:22.811055 11420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:22.811156 11420 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 17:02:22.811187 11420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:22.811239 11420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:22.811276 11420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:22.811306 11420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:02:22.817013 11420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:43945
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:43945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:22.818389 11420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:22.819506 11420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:22.829628 11426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:22.829614 11425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:22.829677 11420 server_base.cc:1034] running on GCE node
W20260806 17:02:22.829629 11428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:22.830678 11420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:22.831990 11420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:22.833249 11420 hybrid_clock.cc:648] HybridClock initialized: now 1786035742833178 us; error 56 us; skew 500 ppm
Aug 06 17:02:22 dist-test-slave-69rg krb5kdc[11406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035742, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:22.836709 11420 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:02:22.837746 11420 webserver.cc:460] Webserver started at http://127.9.111.126:40439/ using document root <none> and password file <none>
I20260806 17:02:22.838439 11420 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:22.838570 11420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:22.838832 11420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:22.840996 11420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "d5887e2cf64941ae8b43e9a19d75d024"
format_stamp: "Formatted at 2026-08-06 17:02:22 on dist-test-slave-69rg"
server_key: "34c4ba3bd74e0be6b89b718124583306"
server_key_iv: "c5f17893fb6a794cceba2e45f59826df"
server_key_version: "encryptionkey@0"
I20260806 17:02:22.841598 11420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "d5887e2cf64941ae8b43e9a19d75d024"
format_stamp: "Formatted at 2026-08-06 17:02:22 on dist-test-slave-69rg"
server_key: "34c4ba3bd74e0be6b89b718124583306"
server_key_iv: "c5f17893fb6a794cceba2e45f59826df"
server_key_version: "encryptionkey@0"
I20260806 17:02:22.845688 11420 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260806 17:02:22.848387 11435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:22.849929 11420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:22.850097 11420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "d5887e2cf64941ae8b43e9a19d75d024"
format_stamp: "Formatted at 2026-08-06 17:02:22 on dist-test-slave-69rg"
server_key: "34c4ba3bd74e0be6b89b718124583306"
server_key_iv: "c5f17893fb6a794cceba2e45f59826df"
server_key_version: "encryptionkey@0"
I20260806 17:02:22.850234 11420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:22.873493 11420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:22.878001 11420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:22.878306 11420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:22.889478 11420 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:43945
I20260806 17:02:22.889541 11500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:43945 every 8 connection(s)
I20260806 17:02:22.890755 11420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:02:22.894593 11501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:22.898154 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 11420
I20260806 17:02:22.898286 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:02:22.898564 9661 external_mini_cluster.cc:1428] Setting key 1eee9011fd6421cc92b15bab0e72192c
I20260806 17:02:22.900892 11501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024: Bootstrap starting.
I20260806 17:02:22.903506 11501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:22.904296 11501 log.cc:824] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:22.906720 11501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024: No bootstrap required, opened a new log
I20260806 17:02:22.911779 11501 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5887e2cf64941ae8b43e9a19d75d024" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43945 } }
Aug 06 17:02:22 dist-test-slave-69rg krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:22.912031 11501 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:22.912096 11501 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5887e2cf64941ae8b43e9a19d75d024, State: Initialized, Role: FOLLOWER
I20260806 17:02:22.912645 11501 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5887e2cf64941ae8b43e9a19d75d024" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43945 } }
I20260806 17:02:22.912825 11501 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:02:22.912878 11501 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:02:22.912978 11501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:22.914278 11501 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5887e2cf64941ae8b43e9a19d75d024" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43945 } }
I20260806 17:02:22.914700 11501 leader_election.cc:302] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5887e2cf64941ae8b43e9a19d75d024; no voters:
I20260806 17:02:22.915112 11501 leader_election.cc:288] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:02:22.915282 11506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:22.915477 11506 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024 [term 1 LEADER]: Becoming Leader. State: Replica: d5887e2cf64941ae8b43e9a19d75d024, State: Running, Role: LEADER
I20260806 17:02:22.915784 11506 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5887e2cf64941ae8b43e9a19d75d024" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43945 } }
I20260806 17:02:22.916505 11501 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:22.917001 11506 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5887e2cf64941ae8b43e9a19d75d024. Latest consensus state: current_term: 1 leader_uuid: "d5887e2cf64941ae8b43e9a19d75d024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5887e2cf64941ae8b43e9a19d75d024" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43945 } } }
I20260806 17:02:22.917120 11506 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:22.917898 11507 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5887e2cf64941ae8b43e9a19d75d024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5887e2cf64941ae8b43e9a19d75d024" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43945 } } }
I20260806 17:02:22.918000 11507 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:22.918303 11510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:22.920058 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:22.900038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53964 (local address 127.9.111.126:43945)
0806 17:02:22.900568 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:22.900593 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:22.902165 (+ 1572us) server_negotiation.cc:407] Connection header received
0806 17:02:22.903365 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:22.903393 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:22.903951 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:22.904367 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:22.905235 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:22.907140 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:22.908354 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:22.909236 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:22.909444 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:22.912754 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:22.912763 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:22.912973 (+ 210us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:22.913022 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:22.915518 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:22.916207 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:22.916212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:22.916217 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:22.916342 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:22.916778 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:22.916782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:22.916784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:22.917493 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:22.918575 (+ 1082us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:22.918761 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 17:02:22.919120 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":153,"threads_started":1}
I20260806 17:02:22.921721 11510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:02:22.927989 11510 catalog_manager.cc:1347] Generated new cluster ID: 353136cc49cf4d6dad00a4f72ae07ddd
I20260806 17:02:22.928082 11510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:22.952886 11510 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:02:22.953997 11510 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:22.959511 11510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024: Generated new TSK 0
I20260806 17:02:22.960371 11510 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
I20260806 17:02:22.997143 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:43945
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:23.122331 11528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:23.122668 11528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:23.122721 11528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:23.127465 11528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:23.127553 11528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:23.127599 11528 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 17:02:23.127614 11528 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 17:02:23.127696 11528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:02:23.133503 11528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.111.126:43945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:23.135191 11528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:23.136439 11528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 17:02:23.147373 11528 server_base.cc:1034] running on GCE node
W20260806 17:02:23.147329 11534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:23.147424 11533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:23.147328 11536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:23.148173 11528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:23.148985 11528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:23.150169 11528 hybrid_clock.cc:648] HybridClock initialized: now 1786035743150149 us; error 36 us; skew 500 ppm
Aug 06 17:02:23 dist-test-slave-69rg krb5kdc[11406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035743, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:23.158689 11528 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:02:23.159834 11528 webserver.cc:460] Webserver started at http://127.9.111.65:40385/ using document root <none> and password file <none>
I20260806 17:02:23.160626 11528 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:23.160780 11528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:23.161149 11528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:23.163656 11528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "c5d8a42e20494b5d8905c8dc0222ad74"
format_stamp: "Formatted at 2026-08-06 17:02:23 on dist-test-slave-69rg"
server_key: "bf42a013de73851be506fd963143fbe1"
server_key_iv: "9e2e77bc626bf4d21c2f93ef41735819"
server_key_version: "encryptionkey@0"
I20260806 17:02:23.164328 11528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "c5d8a42e20494b5d8905c8dc0222ad74"
format_stamp: "Formatted at 2026-08-06 17:02:23 on dist-test-slave-69rg"
server_key: "bf42a013de73851be506fd963143fbe1"
server_key_iv: "9e2e77bc626bf4d21c2f93ef41735819"
server_key_version: "encryptionkey@0"
I20260806 17:02:23.168646 11528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 17:02:23.171449 11544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:23.172847 11528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 17:02:23.173015 11528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "c5d8a42e20494b5d8905c8dc0222ad74"
format_stamp: "Formatted at 2026-08-06 17:02:23 on dist-test-slave-69rg"
server_key: "bf42a013de73851be506fd963143fbe1"
server_key_iv: "9e2e77bc626bf4d21c2f93ef41735819"
server_key_version: "encryptionkey@0"
I20260806 17:02:23.173163 11528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:23.190455 11528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:23.193614 11528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:23.193903 11528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:23.195366 11528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:23.195442 11528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:23.195503 11528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:23.195555 11528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 17:02:23 dist-test-slave-69rg krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035743, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:23.207209 11528 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:41463
I20260806 17:02:23.207288 11664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:41463 every 8 connection(s)
I20260806 17:02:23.208460 11528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:02:23.211664 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.196532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:50337 (local address 127.9.111.126:43945)
0806 17:02:23.196681 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:23.196686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:23.197799 (+ 1113us) server_negotiation.cc:407] Connection header received
0806 17:02:23.199166 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:23.199171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:23.199245 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:23.199359 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:23.200919 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.201913 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.203393 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.203673 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.203801 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.206921 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:23.206925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:23.206928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:23.206988 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:23.209096 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:23.209881 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:23.209886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:23.209888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:23.210001 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:23.210471 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:23.210474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:23.210476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:23.210692 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:23.210792 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:23.211378 (+ 586us) server_negotiation.cc:299] Negotiation successful
0806 17:02:23.211518 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 17:02:23.212585 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.197069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:43945 (local address 127.9.111.65:50337)
0806 17:02:23.197616 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:23.197662 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:23.198836 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:23.199584 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:23.199616 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:23.200091 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:23.200714 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.200729 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.202073 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.202077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.203245 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.203257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.203870 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.203874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.204091 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.204785 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:23.204803 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:23.206701 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:23.209267 (+ 2566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:23.209275 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:23.209291 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:23.209676 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:23.210156 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:23.210159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:23.210161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:23.210338 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:23.210821 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:23.210828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:23.211118 (+ 290us) client_negotiation.cc:769] Sending connection context
0806 17:02:23.211301 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 17:02:23.211590 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":142,"threads_started":1}
I20260806 17:02:23.214377 11665 heartbeater.cc:347] Connected to a master server at 127.9.111.126:43945
I20260806 17:02:23.214560 11665 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:23.215111 11665 heartbeater.cc:511] Master 127.9.111.126:43945 requested a full tablet report, sending...
I20260806 17:02:23.216091 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 11528
I20260806 17:02:23.216261 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:02:23.216599 9661 external_mini_cluster.cc:1428] Setting key 95688a39f459af31cf2cd7bc1b69d1cb
I20260806 17:02:23.217101 11453 ts_manager.cc:194] Registered new tserver with Master: c5d8a42e20494b5d8905c8dc0222ad74 (127.9.111.65:41463)
I20260806 17:02:23.218533 11453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:50337
I20260806 17:02:23.226940 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.216837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:58433 (local address 127.9.111.126:43945)
0806 17:02:23.216966 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:23.216971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:23.217519 (+ 548us) server_negotiation.cc:407] Connection header received
0806 17:02:23.217690 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:23.217693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:23.217746 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:23.217871 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:23.219262 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.219758 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.221254 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.221481 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.221560 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.223192 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:23.223195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:23.223198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:23.223226 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:23.224836 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:23.225420 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:23.225423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:23.225424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:23.225499 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:23.225934 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:23.225938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:23.225940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:23.226220 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:23.226315 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:23.226691 (+ 376us) server_negotiation.cc:299] Negotiation successful
0806 17:02:23.226811 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1270,"server-negotiator.queue_time_us":52}
I20260806 17:02:23.227092 11671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.216791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:43945 (local address 127.9.111.65:58433)
0806 17:02:23.217415 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:23.217428 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:23.217590 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:23.217969 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:23.217972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:23.218242 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:23.218384 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.218393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.219898 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.219906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.221106 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.221122 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.221616 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.221622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.221783 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.222439 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:23.222460 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:23.223034 (+ 574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:23.224969 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:23.224976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:23.224980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:23.225310 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:23.225602 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:23.225608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:23.225612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:23.225781 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:23.226346 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:23.226351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:23.226476 (+ 125us) client_negotiation.cc:769] Sending connection context
0806 17:02:23.226597 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 17:02:23.226795 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":85,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.9.111.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
I20260806 17:02:23.250057 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:43945
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:23.366866 11674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:23.367136 11674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:23.367185 11674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:23.371294 11674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:23.371367 11674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:23.371409 11674 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 17:02:23.371423 11674 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 17:02:23.371507 11674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:02:23.378321 11674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.111.126:43945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:23.379459 11674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:23.380522 11674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:23.390650 11680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:23.390655 11679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:23.390655 11682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:23.391228 11674 server_base.cc:1034] running on GCE node
I20260806 17:02:23.391722 11674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:23.392515 11674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:23.393751 11674 hybrid_clock.cc:648] HybridClock initialized: now 1786035743393706 us; error 50 us; skew 500 ppm
Aug 06 17:02:23 dist-test-slave-69rg krb5kdc[11406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035743, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:23.396850 11674 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:02:23.397796 11674 webserver.cc:460] Webserver started at http://127.9.111.66:45989/ using document root <none> and password file <none>
I20260806 17:02:23.398463 11674 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:23.398561 11674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:23.398810 11674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:23.400904 11674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e"
format_stamp: "Formatted at 2026-08-06 17:02:23 on dist-test-slave-69rg"
server_key: "5ce6d7c08527ac075ed883ce9bd3d56d"
server_key_iv: "cb643cda49daadfaf69bf36f8884a002"
server_key_version: "encryptionkey@0"
I20260806 17:02:23.401501 11674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e"
format_stamp: "Formatted at 2026-08-06 17:02:23 on dist-test-slave-69rg"
server_key: "5ce6d7c08527ac075ed883ce9bd3d56d"
server_key_iv: "cb643cda49daadfaf69bf36f8884a002"
server_key_version: "encryptionkey@0"
I20260806 17:02:23.405535 11674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260806 17:02:23.408285 11689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:23.409596 11674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 17:02:23.409766 11674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e"
format_stamp: "Formatted at 2026-08-06 17:02:23 on dist-test-slave-69rg"
server_key: "5ce6d7c08527ac075ed883ce9bd3d56d"
server_key_iv: "cb643cda49daadfaf69bf36f8884a002"
server_key_version: "encryptionkey@0"
I20260806 17:02:23.409955 11674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:23.430413 11674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:23.434258 11674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:23.434501 11674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:23.436291 11674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:23.436380 11674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:23.436463 11674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:23.436513 11674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 17:02:23 dist-test-slave-69rg krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035743, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:23.448935 11674 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:44123
I20260806 17:02:23.449011 11809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:44123 every 8 connection(s)
I20260806 17:02:23.450243 11674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:02:23.453013 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.437296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:34431 (local address 127.9.111.126:43945)
0806 17:02:23.437503 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:23.437511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:23.438557 (+ 1046us) server_negotiation.cc:407] Connection header received
0806 17:02:23.439753 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:23.439760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:23.439834 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:23.439958 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:23.441742 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.442516 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.444763 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.445227 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.445336 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.448858 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:23.448885 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:23.448890 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:23.448932 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:23.450764 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:23.451403 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:23.451407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:23.451409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:23.451527 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:23.451879 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:23.451882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:23.451883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:23.452091 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:23.452200 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:23.452764 (+ 564us) server_negotiation.cc:299] Negotiation successful
0806 17:02:23.452886 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 17:02:23.454133 11699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.437781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:43945 (local address 127.9.111.66:34431)
0806 17:02:23.438367 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:23.438424 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:23.439475 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:23.440200 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:23.440212 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:23.440730 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:23.441547 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.441574 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.442683 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.442689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.444569 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.444587 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.445392 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.445397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.445629 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.446446 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:23.446475 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:23.448653 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:23.450912 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:23.450921 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:23.450936 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:23.451293 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:23.451635 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:23.451637 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:23.451639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:23.451771 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:23.452241 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:23.452246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:23.452493 (+ 247us) client_negotiation.cc:769] Sending connection context
0806 17:02:23.452661 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 17:02:23.452931 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":155,"threads_started":1}
I20260806 17:02:23.455456 11810 heartbeater.cc:347] Connected to a master server at 127.9.111.126:43945
I20260806 17:02:23.455688 11810 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:23.456233 11810 heartbeater.cc:511] Master 127.9.111.126:43945 requested a full tablet report, sending...
I20260806 17:02:23.457407 11454 ts_manager.cc:194] Registered new tserver with Master: e114e7ba5e8f4ff08f027a7ca2a3d25e (127.9.111.66:44123)
I20260806 17:02:23.458137 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 11674
I20260806 17:02:23.458262 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:02:23.458410 11454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:34431
I20260806 17:02:23.458578 9661 external_mini_cluster.cc:1428] Setting key 76ccfdeaaf0d862d74f2a9e4b1f9ff47
I20260806 17:02:23.467254 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.457904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:45069 (local address 127.9.111.126:43945)
0806 17:02:23.458014 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:23.458018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:23.459436 (+ 1418us) server_negotiation.cc:407] Connection header received
0806 17:02:23.459583 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:23.459585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:23.459649 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:23.459757 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:23.460270 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.460770 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.461961 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.462255 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.462357 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.463905 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:23.463908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:23.463911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:23.463935 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:23.465425 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:23.465921 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:23.465927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:23.465929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:23.466018 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:23.466319 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:23.466323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:23.466325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:23.466571 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:23.466693 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:23.466958 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 17:02:23.467074 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 17:02:23.467264 11816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.457737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:43945 (local address 127.9.111.66:45069)
0806 17:02:23.459168 (+ 1431us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:23.459357 (+ 189us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:23.459482 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:23.459780 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:23.459784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:23.460024 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:23.460148 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.460156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.460906 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.460910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.461752 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.461763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.462397 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.462404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.462547 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.463120 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:23.463143 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:23.463705 (+ 562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:23.465552 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:23.465556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:23.465559 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:23.465781 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:23.466124 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:23.466127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:23.466129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:23.466224 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:23.466677 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:23.466682 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:23.466787 (+ 105us) client_negotiation.cc:769] Sending connection context
0806 17:02:23.466902 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 17:02:23.467068 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1339,"thread_start_us":143,"threads_started":1}
WARNING: no policy specified for kudu/127.9.111.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
I20260806 17:02:23.494436 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:43945
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:23.619381 11819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:23.619663 11819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:23.619714 11819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:23.623829 11819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:23.623890 11819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:23.623930 11819 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 17:02:23.623945 11819 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 17:02:23.623978 11819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:02:23.629154 11819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.111.126:43945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:23.630187 11819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:23.631206 11819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:23.640944 11824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:23.640928 11825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:23.641225 11819 server_base.cc:1034] running on GCE node
W20260806 17:02:23.641350 11827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:23.641813 11819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:23.642529 11819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:23.643723 11819 hybrid_clock.cc:648] HybridClock initialized: now 1786035743643673 us; error 63 us; skew 500 ppm
Aug 06 17:02:23 dist-test-slave-69rg krb5kdc[11406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035743, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:23.646759 11819 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:02:23.647734 11819 webserver.cc:460] Webserver started at http://127.9.111.67:46791/ using document root <none> and password file <none>
I20260806 17:02:23.648320 11819 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:23.648417 11819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:23.648662 11819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:23.651016 11819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "a25f19d5dc774109978ca0c1e7a3609a"
format_stamp: "Formatted at 2026-08-06 17:02:23 on dist-test-slave-69rg"
server_key: "0405a2ed16314ddd4c86ea72f7ae9f42"
server_key_iv: "f8cd185d5046568f5cc6f576670ae438"
server_key_version: "encryptionkey@0"
I20260806 17:02:23.651705 11819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "a25f19d5dc774109978ca0c1e7a3609a"
format_stamp: "Formatted at 2026-08-06 17:02:23 on dist-test-slave-69rg"
server_key: "0405a2ed16314ddd4c86ea72f7ae9f42"
server_key_iv: "f8cd185d5046568f5cc6f576670ae438"
server_key_version: "encryptionkey@0"
I20260806 17:02:23.656769 11819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 17:02:23.659776 11834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:23.661077 11819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 17:02:23.661269 11819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "a25f19d5dc774109978ca0c1e7a3609a"
format_stamp: "Formatted at 2026-08-06 17:02:23 on dist-test-slave-69rg"
server_key: "0405a2ed16314ddd4c86ea72f7ae9f42"
server_key_iv: "f8cd185d5046568f5cc6f576670ae438"
server_key_version: "encryptionkey@0"
I20260806 17:02:23.661408 11819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:23.672623 11819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:23.676020 11819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:23.676242 11819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:23.677856 11819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:23.677933 11819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:23.677997 11819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:23.678048 11819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 17:02:23 dist-test-slave-69rg krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035743, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:23.689136 11819 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:44043
I20260806 17:02:23.689432 11951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:44043 every 8 connection(s)
I20260806 17:02:23.690173 11819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:02:23.692116 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.678800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:33415 (local address 127.9.111.126:43945)
0806 17:02:23.678944 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:23.678952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:23.679967 (+ 1015us) server_negotiation.cc:407] Connection header received
0806 17:02:23.681118 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:23.681122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:23.681176 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:23.681224 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:23.682812 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.683411 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.684659 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.684885 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.685004 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.688221 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:23.688224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:23.688226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:23.688253 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:23.689942 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:23.690504 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:23.690507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:23.690509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:23.690623 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:23.691011 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:23.691015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:23.691016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:23.691255 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:23.691352 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:23.691879 (+ 527us) server_negotiation.cc:299] Negotiation successful
0806 17:02:23.691991 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 17:02:23.692795 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 11819
I20260806 17:02:23.692950 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:02:23.693104 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.679308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:43945 (local address 127.9.111.67:33415)
0806 17:02:23.679808 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:23.679850 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:23.680836 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:23.681596 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:23.681602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:23.682060 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:23.682634 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.682658 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.683554 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.683557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.684527 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.684537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.685056 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.685059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.685233 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.685926 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:23.685942 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:23.688035 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:23.690058 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:23.690064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:23.690077 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:23.690394 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:23.690739 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:23.690742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:23.690744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:23.690912 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:23.691391 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:23.691396 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:23.691644 (+ 248us) client_negotiation.cc:769] Sending connection context
0806 17:02:23.691822 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 17:02:23.692078 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":141,"threads_started":1}
I20260806 17:02:23.693347 9661 external_mini_cluster.cc:1428] Setting key 2e2f88c73c1b67f766acc058dd84b568
I20260806 17:02:23.694648 11952 heartbeater.cc:347] Connected to a master server at 127.9.111.126:43945
I20260806 17:02:23.694842 11952 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:23.695461 11952 heartbeater.cc:511] Master 127.9.111.126:43945 requested a full tablet report, sending...
I20260806 17:02:23.696525 11454 ts_manager.cc:194] Registered new tserver with Master: a25f19d5dc774109978ca0c1e7a3609a (127.9.111.67:44043)
I20260806 17:02:23.697319 11454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:33415
I20260806 17:02:23.704352 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.697054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:53869 (local address 127.9.111.126:43945)
0806 17:02:23.697178 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:23.697188 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:23.697405 (+ 217us) server_negotiation.cc:407] Connection header received
0806 17:02:23.697566 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:23.697569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:23.697621 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:23.697702 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:23.698259 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.698784 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.699634 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.699857 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.699968 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.701124 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:23.701127 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:23.701129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:23.701152 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:23.702585 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:23.703087 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:23.703090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:23.703092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:23.703175 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:23.703472 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:23.703477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:23.703478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:23.703666 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:23.703787 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:23.704061 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 17:02:23.704178 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 17:02:23.704375 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.696966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:43945 (local address 127.9.111.67:53869)
0806 17:02:23.697316 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:23.697328 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:23.697448 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:23.697727 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:23.697730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:23.697986 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:23.698131 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.698138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.698930 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.698933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.699504 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.699512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.699973 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.699977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.700070 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.700598 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:23.700610 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:23.700952 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:23.702759 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:23.702762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:23.702765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:23.702974 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:23.703283 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:23.703287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:23.703289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:23.703379 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:23.703775 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:23.703779 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:23.703894 (+ 115us) client_negotiation.cc:769] Sending connection context
0806 17:02:23.704004 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 17:02:23.704190 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260806 17:02:23.708204 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 17:02:23 dist-test-slave-69rg krb5kdc[11406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 17:02:23 dist-test-slave-69rg krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:23.738358 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.728885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53980 (local address 127.9.111.126:43945)
0806 17:02:23.729034 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:23.729038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:23.729051 (+ 13us) server_negotiation.cc:407] Connection header received
0806 17:02:23.729094 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:23.729145 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:23.729187 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:23.729295 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:23.730157 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.730686 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.731552 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.731794 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.731905 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.734685 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:23.734689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:23.734692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:23.734752 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:23.736460 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:23.736961 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:23.736966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:23.736968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:23.737060 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:23.737379 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:23.737383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:23.737385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:23.737590 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:23.737740 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:23.738028 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 17:02:23.738180 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 17:02:23.742213 11454 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:02:23.745724 11454 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:02:23.762460 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.756296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49464 (local address 127.9.111.65:41463)
0806 17:02:23.756656 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:23.756662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:23.757212 (+ 550us) server_negotiation.cc:407] Connection header received
0806 17:02:23.757294 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:23.757298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:23.757450 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:23.757571 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:23.758110 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.759036 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.760871 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.761656 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.761744 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.761940 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:23.762192 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 17:02:23.762303 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":103,"threads_started":1}
I20260806 17:02:23.762482 11969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.755517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:41463 (local address 127.0.0.1:49464)
0806 17:02:23.756801 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:23.756862 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:23.757005 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:23.757607 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:23.757613 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:23.757637 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:23.757969 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.757981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.759159 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.759165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.760737 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.760744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.761790 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.761794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.761965 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.762011 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 17:02:23.762141 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 17:02:23.762303 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"mutex_wait_us":138,"spinlock_wait_cycles":1792,"thread_start_us":111,"threads_started":1}
I20260806 17:02:23.762900 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.756316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:44123 (local address 127.0.0.1:36714)
0806 17:02:23.757224 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:23.757238 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:23.757369 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:23.758433 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:23.758436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:23.758455 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:23.758544 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.758550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.760239 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.760244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.761243 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.761254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.762467 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.762470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.762567 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.762601 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 17:02:23.762688 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 17:02:23.762777 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"thread_start_us":94,"threads_started":1}
I20260806 17:02:23.762996 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.756864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36714 (local address 127.9.111.66:44123)
0806 17:02:23.757952 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:23.757964 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:23.757988 (+ 24us) server_negotiation.cc:407] Connection header received
0806 17:02:23.758061 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:23.758069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:23.758281 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:23.758433 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:23.758666 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.760124 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.761383 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.762160 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.762255 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.762402 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:23.762745 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 17:02:23.762854 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":945,"thread_start_us":119,"threads_started":1}
I20260806 17:02:23.764171 11973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.755576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:44043 (local address 127.0.0.1:37764)
0806 17:02:23.756915 (+ 1339us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:23.756929 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:23.757005 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:23.757660 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:23.757664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:23.757682 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:23.757964 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.757976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.759429 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.759432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.760731 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.760744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.763758 (+ 3014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.763762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.763864 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.763888 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 17:02:23.763970 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 17:02:23.764046 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1266,"mutex_wait_us":102,"thread_start_us":104,"threads_started":1}
I20260806 17:02:23.764271 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.755810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.9.111.67:44043)
0806 17:02:23.756495 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:23.756507 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:23.757150 (+ 643us) server_negotiation.cc:407] Connection header received
0806 17:02:23.757240 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:23.757246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:23.757472 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:23.757652 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:23.758134 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.759296 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.761932 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.763630 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.763742 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.763891 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:23.763993 (+ 102us) server_negotiation.cc:299] Negotiation successful
0806 17:02:23.764089 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":131,"threads_started":1}
I20260806 17:02:23.766045 11597 tablet_service.cc:1467] Processing CreateTablet for tablet fc058498ee8f432c94ecd2caadca6120 (DEFAULT_TABLE table=test-table [id=8775289c8c2d4c8da053122ce3c371fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:23.767266 11597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc058498ee8f432c94ecd2caadca6120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:23.767855 11741 tablet_service.cc:1467] Processing CreateTablet for tablet fc058498ee8f432c94ecd2caadca6120 (DEFAULT_TABLE table=test-table [id=8775289c8c2d4c8da053122ce3c371fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:23.768642 11741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc058498ee8f432c94ecd2caadca6120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:23.768412 11885 tablet_service.cc:1467] Processing CreateTablet for tablet fc058498ee8f432c94ecd2caadca6120 (DEFAULT_TABLE table=test-table [id=8775289c8c2d4c8da053122ce3c371fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:23.769418 11885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc058498ee8f432c94ecd2caadca6120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:23.775983 11975 tablet_bootstrap.cc:492] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Bootstrap starting.
I20260806 17:02:23.776089 11976 tablet_bootstrap.cc:492] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74: Bootstrap starting.
I20260806 17:02:23.776873 11977 tablet_bootstrap.cc:492] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a: Bootstrap starting.
I20260806 17:02:23.778368 11976 tablet_bootstrap.cc:654] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:23.778427 11975 tablet_bootstrap.cc:654] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:23.779140 11976 log.cc:824] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:23.779229 11975 log.cc:824] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:23.779389 11977 tablet_bootstrap.cc:654] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:23.780064 11977 log.cc:824] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:23.781288 11976 tablet_bootstrap.cc:492] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74: No bootstrap required, opened a new log
I20260806 17:02:23.781517 11976 ts_tablet_manager.cc:1397] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 17:02:23.781790 11975 tablet_bootstrap.cc:492] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e: No bootstrap required, opened a new log
I20260806 17:02:23.782014 11975 ts_tablet_manager.cc:1397] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 17:02:23.782081 11977 tablet_bootstrap.cc:492] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a: No bootstrap required, opened a new log
I20260806 17:02:23.782248 11977 ts_tablet_manager.cc:1397] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 17:02:23.785161 11975 raft_consensus.cc:348] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } }
I20260806 17:02:23.785465 11975 raft_consensus.cc:374] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:23.785523 11975 raft_consensus.cc:729] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e114e7ba5e8f4ff08f027a7ca2a3d25e, State: Initialized, Role: FOLLOWER
I20260806 17:02:23.786011 11975 consensus_queue.cc:260] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } }
I20260806 17:02:23.786314 11976 raft_consensus.cc:348] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } }
I20260806 17:02:23.786314 11977 raft_consensus.cc:348] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } }
I20260806 17:02:23.786568 11977 raft_consensus.cc:374] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:23.786568 11976 raft_consensus.cc:374] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:23.786630 11977 raft_consensus.cc:729] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a25f19d5dc774109978ca0c1e7a3609a, State: Initialized, Role: FOLLOWER
I20260806 17:02:23.786630 11976 raft_consensus.cc:729] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d8a42e20494b5d8905c8dc0222ad74, State: Initialized, Role: FOLLOWER
I20260806 17:02:23.786890 11810 heartbeater.cc:503] Master 127.9.111.126:43945 was elected leader, sending a full tablet report...
I20260806 17:02:23.787087 11976 consensus_queue.cc:260] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } }
I20260806 17:02:23.787087 11977 consensus_queue.cc:260] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } }
I20260806 17:02:23.787343 11975 ts_tablet_manager.cc:1428] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 17:02:23.788009 11952 heartbeater.cc:503] Master 127.9.111.126:43945 was elected leader, sending a full tablet report...
I20260806 17:02:23.788040 11977 ts_tablet_manager.cc:1428] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 17:02:23.788040 11976 ts_tablet_manager.cc:1428] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 17:02:23.789078 11665 heartbeater.cc:503] Master 127.9.111.126:43945 was elected leader, sending a full tablet report...
I20260806 17:02:23.850481 11982 raft_consensus.cc:482] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:23.850709 11982 raft_consensus.cc:504] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } }
I20260806 17:02:23.851976 11982 leader_election.cc:288] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a25f19d5dc774109978ca0c1e7a3609a (127.9.111.67:44043), e114e7ba5e8f4ff08f027a7ca2a3d25e (127.9.111.66:44123)
I20260806 17:02:23.857232 11984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.852246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:44123 (local address 127.9.111.65:56047)
0806 17:02:23.852767 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:23.852788 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:23.852950 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:23.853295 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:23.853299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:23.853348 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:23.853677 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.853683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.854635 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.854641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.855892 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.855904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.856715 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.856719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.856852 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.856880 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 17:02:23.856955 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 17:02:23.857030 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":64,"spinlock_wait_cycles":33280,"thread_start_us":132,"threads_started":1}
I20260806 17:02:23.857230 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.852401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:56047 (local address 127.9.111.66:44123)
0806 17:02:23.852562 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:23.852568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:23.852928 (+ 360us) server_negotiation.cc:407] Connection header received
0806 17:02:23.853073 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:23.853077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:23.853131 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:23.853225 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:23.853876 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.854511 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.856022 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.856559 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.856702 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.856776 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:23.856990 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 17:02:23.857088 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 17:02:23.857362 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.852474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:38927 (local address 127.9.111.67:44043)
0806 17:02:23.852651 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:23.852660 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:23.853042 (+ 382us) server_negotiation.cc:407] Connection header received
0806 17:02:23.853100 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:23.853102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:23.853148 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:23.853225 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:23.853875 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.854474 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.856101 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.856718 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.856826 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.856888 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:23.857141 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 17:02:23.857219 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 17:02:23.857297 11985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.852327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:44043 (local address 127.9.111.65:38927)
0806 17:02:23.852871 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:23.852885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:23.852969 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:23.853295 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:23.853299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:23.853349 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:23.853657 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.853668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.854609 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.854614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.855931 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.855943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.856899 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.856905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.857008 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.857025 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 17:02:23.857088 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 17:02:23.857143 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":136,"spinlock_wait_cycles":10496,"thread_start_us":129,"threads_started":1}
I20260806 17:02:23.858170 11761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc058498ee8f432c94ecd2caadca6120" candidate_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" is_pre_election: true
I20260806 17:02:23.858429 11905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc058498ee8f432c94ecd2caadca6120" candidate_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a25f19d5dc774109978ca0c1e7a3609a" is_pre_election: true
I20260806 17:02:23.858526 11761 raft_consensus.cc:2454] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5d8a42e20494b5d8905c8dc0222ad74 in term 0.
I20260806 17:02:23.858729 11905 raft_consensus.cc:2454] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5d8a42e20494b5d8905c8dc0222ad74 in term 0.
I20260806 17:02:23.859126 11548 leader_election.cc:302] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a25f19d5dc774109978ca0c1e7a3609a, c5d8a42e20494b5d8905c8dc0222ad74, e114e7ba5e8f4ff08f027a7ca2a3d25e; no voters:
I20260806 17:02:23.859431 11982 raft_consensus.cc:2781] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:02:23.859524 11982 raft_consensus.cc:482] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:02:23.859581 11982 raft_consensus.cc:3037] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:23.860739 11982 raft_consensus.cc:504] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } }
I20260806 17:02:23.861140 11982 leader_election.cc:288] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [CANDIDATE]: Term 1 election: Requested vote from peers a25f19d5dc774109978ca0c1e7a3609a (127.9.111.67:44043), e114e7ba5e8f4ff08f027a7ca2a3d25e (127.9.111.66:44123)
I20260806 17:02:23.861557 11761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc058498ee8f432c94ecd2caadca6120" candidate_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e"
I20260806 17:02:23.861557 11905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc058498ee8f432c94ecd2caadca6120" candidate_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a25f19d5dc774109978ca0c1e7a3609a"
I20260806 17:02:23.861708 11761 raft_consensus.cc:3037] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:23.861708 11905 raft_consensus.cc:3037] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:23.863142 11761 raft_consensus.cc:2454] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5d8a42e20494b5d8905c8dc0222ad74 in term 1.
I20260806 17:02:23.863195 11905 raft_consensus.cc:2454] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5d8a42e20494b5d8905c8dc0222ad74 in term 1.
I20260806 17:02:23.863539 11548 leader_election.cc:302] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5d8a42e20494b5d8905c8dc0222ad74, e114e7ba5e8f4ff08f027a7ca2a3d25e; no voters:
I20260806 17:02:23.863843 11982 raft_consensus.cc:2781] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:23.864071 11982 raft_consensus.cc:686] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [term 1 LEADER]: Becoming Leader. State: Replica: c5d8a42e20494b5d8905c8dc0222ad74, State: Running, Role: LEADER
I20260806 17:02:23.864444 11982 consensus_queue.cc:237] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } }
I20260806 17:02:23.867898 11454 catalog_manager.cc:5495] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5d8a42e20494b5d8905c8dc0222ad74 (127.9.111.65). New cstate: current_term: 1 leader_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } health_report { overall_health: UNKNOWN } } }
Aug 06 17:02:23 dist-test-slave-69rg krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035742, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:23.912287 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.903532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.9.111.126:43945)
0806 17:02:23.903694 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:23.903700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:23.903869 (+ 169us) server_negotiation.cc:407] Connection header received
0806 17:02:23.903974 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:23.903976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:23.904020 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:23.904094 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:23.904805 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.905369 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.906191 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.906416 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.906502 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.908957 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:23.908960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:23.908962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:23.908986 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:23.910518 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:23.910987 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:23.910991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:23.910992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:23.911082 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:23.911410 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:23.911413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:23.911414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:23.911648 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:23.911735 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:23.912013 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 17:02:23.912124 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 17:02:23.912292 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:23.903442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:43945 (local address 127.0.0.1:53992)
0806 17:02:23.903772 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:23.903785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:23.903888 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:23.904133 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:23.904134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:23.904565 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:23.904668 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.904674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.905488 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.905491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.906081 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:23.906090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:23.906520 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:23.906523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:23.906584 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:23.907168 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:23.907182 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:23.908827 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:23.910621 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:23.910630 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:23.910635 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:23.910866 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:23.911195 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:23.911197 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:23.911199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:23.911279 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:23.911746 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:23.911752 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:23.911858 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 17:02:23.911948 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 17:02:23.912066 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":149,"threads_started":1}
I20260806 17:02:23.916759 11454 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:53992:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 17:02:23.917524 11454 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:02:23.922991 11597 tablet_service.cc:1467] Processing CreateTablet for tablet 4f559521fa9d41b7818fddc01054837c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=380dbc1a1fad45b3933d8cdae171b6f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 17:02:23.923009 11741 tablet_service.cc:1467] Processing CreateTablet for tablet 4f559521fa9d41b7818fddc01054837c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=380dbc1a1fad45b3933d8cdae171b6f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 17:02:23.923830 11741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f559521fa9d41b7818fddc01054837c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:23.923830 11597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f559521fa9d41b7818fddc01054837c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:23.923918 11885 tablet_service.cc:1467] Processing CreateTablet for tablet 4f559521fa9d41b7818fddc01054837c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=380dbc1a1fad45b3933d8cdae171b6f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 17:02:23.924536 11885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f559521fa9d41b7818fddc01054837c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:23.927388 11976 tablet_bootstrap.cc:492] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74: Bootstrap starting.
I20260806 17:02:23.927541 11977 tablet_bootstrap.cc:492] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a: Bootstrap starting.
I20260806 17:02:23.928573 11976 tablet_bootstrap.cc:654] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:23.928573 11977 tablet_bootstrap.cc:654] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:23.929786 11975 tablet_bootstrap.cc:492] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e: Bootstrap starting.
I20260806 17:02:23.930204 11977 tablet_bootstrap.cc:492] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a: No bootstrap required, opened a new log
I20260806 17:02:23.930289 11977 ts_tablet_manager.cc:1397] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 17:02:23.930393 11976 tablet_bootstrap.cc:492] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74: No bootstrap required, opened a new log
I20260806 17:02:23.930459 11976 ts_tablet_manager.cc:1397] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 17:02:23.930840 11976 raft_consensus.cc:348] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:23.930795 11977 raft_consensus.cc:348] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:23.930930 11976 raft_consensus.cc:374] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:23.930953 11976 raft_consensus.cc:729] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d8a42e20494b5d8905c8dc0222ad74, State: Initialized, Role: FOLLOWER
I20260806 17:02:23.930958 11977 raft_consensus.cc:374] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:23.931002 11977 raft_consensus.cc:729] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a25f19d5dc774109978ca0c1e7a3609a, State: Initialized, Role: FOLLOWER
I20260806 17:02:23.931044 11975 tablet_bootstrap.cc:654] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:23.931100 11976 consensus_queue.cc:260] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:23.931203 11977 consensus_queue.cc:260] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:23.931339 11976 ts_tablet_manager.cc:1428] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 17:02:23.931511 11977 ts_tablet_manager.cc:1428] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 17:02:23.931530 11983 tablet_replica.cc:439] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } } }
I20260806 17:02:23.931690 11983 tablet_replica.cc:442] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:23.931731 11982 tablet_replica.cc:439] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } } }
I20260806 17:02:23.931881 11982 tablet_replica.cc:442] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:23.933173 11975 tablet_bootstrap.cc:492] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e: No bootstrap required, opened a new log
I20260806 17:02:23.933301 11975 ts_tablet_manager.cc:1397] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 17:02:23.933893 11975 raft_consensus.cc:348] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:23.934048 11975 raft_consensus.cc:374] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:23.934098 11975 raft_consensus.cc:729] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e114e7ba5e8f4ff08f027a7ca2a3d25e, State: Initialized, Role: FOLLOWER
I20260806 17:02:23.934252 11975 consensus_queue.cc:260] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:23.934584 11981 tablet_replica.cc:439] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } } }
I20260806 17:02:23.934772 11981 tablet_replica.cc:442] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:23.935010 11975 ts_tablet_manager.cc:1428] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260806 17:02:23.941238 11953 tablet.cc:2369] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 17:02:23.951529 11811 tablet.cc:2369] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 17:02:23.959980 11666 tablet.cc:2369] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:02:24.201520 11981 raft_consensus.cc:482] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:24.201704 11981 raft_consensus.cc:504] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:24.202963 11981 leader_election.cc:288] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5d8a42e20494b5d8905c8dc0222ad74 (127.9.111.65:41463), a25f19d5dc774109978ca0c1e7a3609a (127.9.111.67:44043)
I20260806 17:02:24.207322 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.203398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:35415 (local address 127.9.111.67:44043)
0806 17:02:24.203705 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:24.203711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:24.203735 (+ 24us) server_negotiation.cc:407] Connection header received
0806 17:02:24.203935 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:24.203939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:24.204001 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:24.204063 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:24.204386 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.205003 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.206286 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.206771 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.206906 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.206981 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:24.207126 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 17:02:24.207208 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260806 17:02:24.207326 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.203294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:44043 (local address 127.9.111.66:35415)
0806 17:02:24.203629 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:24.203647 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:24.203775 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:24.204103 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:24.204108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:24.204130 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:24.204260 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:24.204266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.205130 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.205134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:24.206157 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:24.206168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.206903 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.206908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:24.207005 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.207028 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 17:02:24.207105 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 17:02:24.207184 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":98,"threads_started":1}
I20260806 17:02:24.207551 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.203523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:58659 (local address 127.9.111.65:41463)
0806 17:02:24.203651 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:24.203656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:24.203815 (+ 159us) server_negotiation.cc:407] Connection header received
0806 17:02:24.203988 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:24.203991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:24.204034 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:24.204138 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:24.204489 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.205148 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.206406 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.206861 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.206981 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.207031 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:24.207343 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 17:02:24.207430 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 17:02:24.207933 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.203294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:41463 (local address 127.9.111.66:58659)
0806 17:02:24.203707 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:24.203724 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:24.203840 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:24.204174 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:24.204179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:24.204199 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:24.204312 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:24.204340 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.205309 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.205312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:24.206273 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:24.206284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.207065 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.207069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:24.207183 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.207206 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 17:02:24.207715 (+ 509us) client_negotiation.cc:240] Negotiation successful
0806 17:02:24.207795 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":143,"thread_start_us":59,"threads_started":1}
I20260806 17:02:24.208276 11905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f559521fa9d41b7818fddc01054837c" candidate_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a25f19d5dc774109978ca0c1e7a3609a" is_pre_election: true
I20260806 17:02:24.208467 11905 raft_consensus.cc:2454] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e114e7ba5e8f4ff08f027a7ca2a3d25e in term 0.
I20260806 17:02:24.208576 11618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f559521fa9d41b7818fddc01054837c" candidate_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" is_pre_election: true
I20260806 17:02:24.208810 11618 raft_consensus.cc:2454] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e114e7ba5e8f4ff08f027a7ca2a3d25e in term 0.
I20260806 17:02:24.208881 11693 leader_election.cc:302] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a25f19d5dc774109978ca0c1e7a3609a, e114e7ba5e8f4ff08f027a7ca2a3d25e; no voters:
I20260806 17:02:24.209300 11981 raft_consensus.cc:2781] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:02:24.209371 11981 raft_consensus.cc:482] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:02:24.209412 11981 raft_consensus.cc:3037] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:24.210811 11981 raft_consensus.cc:504] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:24.211246 11981 leader_election.cc:288] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e [CANDIDATE]: Term 1 election: Requested vote from peers c5d8a42e20494b5d8905c8dc0222ad74 (127.9.111.65:41463), a25f19d5dc774109978ca0c1e7a3609a (127.9.111.67:44043)
I20260806 17:02:24.211658 11618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f559521fa9d41b7818fddc01054837c" candidate_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5d8a42e20494b5d8905c8dc0222ad74"
I20260806 17:02:24.211663 11905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f559521fa9d41b7818fddc01054837c" candidate_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a25f19d5dc774109978ca0c1e7a3609a"
I20260806 17:02:24.211795 11618 raft_consensus.cc:3037] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:24.211798 11905 raft_consensus.cc:3037] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:24.212832 11618 raft_consensus.cc:2454] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e114e7ba5e8f4ff08f027a7ca2a3d25e in term 1.
I20260806 17:02:24.213034 11905 raft_consensus.cc:2454] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e114e7ba5e8f4ff08f027a7ca2a3d25e in term 1.
I20260806 17:02:24.213182 11692 leader_election.cc:302] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5d8a42e20494b5d8905c8dc0222ad74, e114e7ba5e8f4ff08f027a7ca2a3d25e; no voters:
I20260806 17:02:24.213419 11981 raft_consensus.cc:2781] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:24.213682 11981 raft_consensus.cc:686] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 LEADER]: Becoming Leader. State: Replica: e114e7ba5e8f4ff08f027a7ca2a3d25e, State: Running, Role: LEADER
I20260806 17:02:24.214102 11981 consensus_queue.cc:237] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:24.216533 11999 tablet_replica.cc:439] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e: TxnStatusTablet state changed. Reason: New leader e114e7ba5e8f4ff08f027a7ca2a3d25e. Latest consensus state: current_term: 1 leader_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } } }
I20260806 17:02:24.216755 11999 tablet_replica.cc:442] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e: This TxnStatusTablet replica's current role is: LEADER
I20260806 17:02:24.217505 11453 catalog_manager.cc:5495] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e reported cstate change: term changed from 0 to 1, leader changed from <none> to e114e7ba5e8f4ff08f027a7ca2a3d25e (127.9.111.66). New cstate: current_term: 1 leader_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } health_report { overall_health: UNKNOWN } } }
I20260806 17:02:24.217973 12001 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 17:02:24.218195 12001 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 17:02:24.267030 11982 consensus_queue.cc:1035] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:24.273475 12005 consensus_queue.cc:1035] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:24.297906 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.293517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:44123 (local address 127.0.0.1:36730)
0806 17:02:24.293700 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:24.293722 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:24.293895 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:24.294299 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:24.294302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:24.294311 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 17:02:24.294439 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:24.294445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.295319 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.295323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:24.296008 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:24.296018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.296539 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.296543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:24.296655 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.296707 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 17:02:24.297512 (+ 805us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 17:02:24.297533 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 17:02:24.297613 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 17:02:24.297724 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 17:02:24.297874 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.293675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36730 (local address 127.9.111.66:44123)
0806 17:02:24.293853 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:24.293859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:24.293876 (+ 17us) server_negotiation.cc:407] Connection header received
0806 17:02:24.294091 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:24.294094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:24.294165 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:24.294281 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:24.294562 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.295197 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.296156 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.296405 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.296522 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.296884 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:24.297398 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:24.297541 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:24.297626 (+ 85us) server_negotiation.cc:299] Negotiation successful
0806 17:02:24.297686 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 17:02:24.303675 11905 raft_consensus.cc:1261] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a [term 1 FOLLOWER]: Refusing update from remote peer e114e7ba5e8f4ff08f027a7ca2a3d25e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:24.303920 11618 raft_consensus.cc:1261] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74 [term 1 FOLLOWER]: Refusing update from remote peer e114e7ba5e8f4ff08f027a7ca2a3d25e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:24.304267 11981 consensus_queue.cc:1035] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:24.304527 11999 consensus_queue.cc:1035] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:24.307502 11983 tablet_replica.cc:439] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a: TxnStatusTablet state changed. Reason: New leader e114e7ba5e8f4ff08f027a7ca2a3d25e. Latest consensus state: current_term: 1 leader_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } } }
I20260806 17:02:24.307643 11983 tablet_replica.cc:442] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:24.307716 12005 tablet_replica.cc:439] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74: TxnStatusTablet state changed. Reason: New leader e114e7ba5e8f4ff08f027a7ca2a3d25e. Latest consensus state: current_term: 1 leader_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } } }
I20260806 17:02:24.307834 12005 tablet_replica.cc:442] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:24.311297 12008 mvcc.cc:205] Tried to move back new op lower bound from 7315602408661114880 to 7315602408304267264. Current Snapshot: MvccSnapshot[applied={T|T < 7315602408661114880}]
I20260806 17:02:24.311506 11981 tablet_replica.cc:439] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } } }
I20260806 17:02:24.311640 11981 tablet_replica.cc:442] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e: This TxnStatusTablet replica's current role is: LEADER
I20260806 17:02:24.312379 11983 tablet_replica.cc:439] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } } }
I20260806 17:02:24.312508 11983 tablet_replica.cc:442] T 4f559521fa9d41b7818fddc01054837c P a25f19d5dc774109978ca0c1e7a3609a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:24.313122 11999 tablet_replica.cc:439] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } } }
I20260806 17:02:24.313232 11999 tablet_replica.cc:442] T 4f559521fa9d41b7818fddc01054837c P e114e7ba5e8f4ff08f027a7ca2a3d25e: This TxnStatusTablet replica's current role is: LEADER
I20260806 17:02:24.314409 12004 mvcc.cc:205] Tried to move back new op lower bound from 7315602408661114880 to 7315602408304267264. Current Snapshot: MvccSnapshot[applied={T|T < 7315602408661114880}]
I20260806 17:02:24.314639 12005 tablet_replica.cc:439] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } } }
I20260806 17:02:24.314766 12005 tablet_replica.cc:442] T 4f559521fa9d41b7818fddc01054837c P c5d8a42e20494b5d8905c8dc0222ad74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:24.326351 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.322002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.9.111.65:41463)
0806 17:02:24.322145 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:24.322150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:24.322390 (+ 240us) server_negotiation.cc:407] Connection header received
0806 17:02:24.322600 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:24.322604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:24.322672 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:24.322799 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:24.323093 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.323660 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.324650 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.324915 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.325039 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.325367 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:24.325933 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:24.326067 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:24.326135 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 17:02:24.326197 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 17:02:24.339594 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.335402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:38047 (local address 127.9.111.66:44123)
0806 17:02:24.335599 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:24.335603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:24.335737 (+ 134us) server_negotiation.cc:407] Connection header received
0806 17:02:24.335892 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:24.335894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:24.335941 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:24.336010 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:24.336352 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.336915 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.338375 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.338610 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.338715 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.339119 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:24.339192 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:24.339282 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:24.339430 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 17:02:24.339479 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260806 17:02:24.339768 12019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.335238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:44123 (local address 127.9.111.65:38047)
0806 17:02:24.335630 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:24.335649 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:24.335780 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:24.336054 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:24.336057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:24.336067 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 17:02:24.336232 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:24.336239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.337031 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.337035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:24.338233 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:24.338246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.338764 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.338768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:24.338879 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.338941 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 17:02:24.339303 (+ 362us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 17:02:24.339314 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 17:02:24.339414 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 17:02:24.339534 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":130,"threads_started":1}
I20260806 17:02:24.369118 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.364962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:54099 (local address 127.9.111.65:41463)
0806 17:02:24.365099 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:24.365104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:24.365321 (+ 217us) server_negotiation.cc:407] Connection header received
0806 17:02:24.365486 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:24.365489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:24.365534 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:24.365708 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:24.366094 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.366691 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.367954 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.368163 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.368238 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.368592 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:24.368695 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:24.368772 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:24.368963 (+ 191us) server_negotiation.cc:299] Negotiation successful
0806 17:02:24.369014 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 17:02:24.369180 12022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.364891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:41463 (local address 127.9.111.66:54099)
0806 17:02:24.365229 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:24.365246 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:24.365355 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:24.365713 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:24.365717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:24.365725 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 17:02:24.365931 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:24.365941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.366814 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.366818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:24.367827 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:24.367838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.368278 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.368281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:24.368373 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.368428 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 17:02:24.368837 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 17:02:24.368845 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 17:02:24.368922 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 17:02:24.369007 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":127,"threads_started":1}
I20260806 17:02:24.398577 11454 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:24.398816 11454 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:24.401678 11454 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024: Sending DeleteTablet for 3 replicas of tablet fc058498ee8f432c94ecd2caadca6120
I20260806 17:02:24.402529 11597 tablet_service.cc:1514] Processing DeleteTablet for tablet fc058498ee8f432c94ecd2caadca6120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:24 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:49464
I20260806 17:02:24.402585 11885 tablet_service.cc:1514] Processing DeleteTablet for tablet fc058498ee8f432c94ecd2caadca6120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:24 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:37764
I20260806 17:02:24.403158 12026 tablet_replica.cc:330] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74: stopping tablet replica
I20260806 17:02:24.403505 12026 raft_consensus.cc:2229] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:02:24.403468 11741 tablet_service.cc:1514] Processing DeleteTablet for tablet fc058498ee8f432c94ecd2caadca6120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:24 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:36714
I20260806 17:02:24.403962 12026 raft_consensus.cc:2258] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:24.404985 11453 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:02:24.405443 11453 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:02:24.405568 12026 ts_tablet_manager.cc:1905] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:24.408671 12026 ts_tablet_manager.cc:1918] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 17:02:24.408905 12026 log.cc:1196] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/wal/wals/fc058498ee8f432c94ecd2caadca6120
I20260806 17:02:24.409173 12028 tablet_replica.cc:330] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e: stopping tablet replica
I20260806 17:02:24.409595 12026 ts_tablet_manager.cc:1939] T fc058498ee8f432c94ecd2caadca6120 P c5d8a42e20494b5d8905c8dc0222ad74: Deleting consensus metadata
I20260806 17:02:24.410000 12028 raft_consensus.cc:2229] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:24.410706 11597 tablet_service.cc:1467] Processing CreateTablet for tablet 4c1c194299fd412ab1d7529a9e43a456 (DEFAULT_TABLE table=test-table [id=028b9b3a1b154089aa28795286c66b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:24.410975 11597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c1c194299fd412ab1d7529a9e43a456. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:24.411002 12028 raft_consensus.cc:2258] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:24.412343 12028 ts_tablet_manager.cc:1905] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:24.413138 11437 catalog_manager.cc:4841] TS c5d8a42e20494b5d8905c8dc0222ad74 (127.9.111.65:41463): tablet fc058498ee8f432c94ecd2caadca6120 (table test-table [id=8775289c8c2d4c8da053122ce3c371fc]) successfully deleted
I20260806 17:02:24.413415 11976 tablet_bootstrap.cc:492] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74: Bootstrap starting.
I20260806 17:02:24.413775 11885 tablet_service.cc:1467] Processing CreateTablet for tablet 4c1c194299fd412ab1d7529a9e43a456 (DEFAULT_TABLE table=test-table [id=028b9b3a1b154089aa28795286c66b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:24.414278 11885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c1c194299fd412ab1d7529a9e43a456. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:24.414419 11976 tablet_bootstrap.cc:654] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:24.415889 11976 tablet_bootstrap.cc:492] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74: No bootstrap required, opened a new log
I20260806 17:02:24.415988 11976 ts_tablet_manager.cc:1397] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 17:02:24.416433 11976 raft_consensus.cc:348] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:24.416567 11976 raft_consensus.cc:374] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:24.416612 11976 raft_consensus.cc:729] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d8a42e20494b5d8905c8dc0222ad74, State: Initialized, Role: FOLLOWER
I20260806 17:02:24.416742 11977 tablet_bootstrap.cc:492] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a: Bootstrap starting.
I20260806 17:02:24.416761 11976 consensus_queue.cc:260] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:24.417114 11976 ts_tablet_manager.cc:1428] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 17:02:24.417153 12028 ts_tablet_manager.cc:1918] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 17:02:24.417352 12028 log.cc:1196] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/fc058498ee8f432c94ecd2caadca6120
I20260806 17:02:24.417709 11977 tablet_bootstrap.cc:654] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:24.417686 12028 ts_tablet_manager.cc:1939] T fc058498ee8f432c94ecd2caadca6120 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Deleting consensus metadata
I20260806 17:02:24.419075 11977 tablet_bootstrap.cc:492] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a: No bootstrap required, opened a new log
I20260806 17:02:24.419175 11977 ts_tablet_manager.cc:1397] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 17:02:24.419131 11741 tablet_service.cc:1467] Processing CreateTablet for tablet 4c1c194299fd412ab1d7529a9e43a456 (DEFAULT_TABLE table=test-table [id=028b9b3a1b154089aa28795286c66b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:24.419368 11741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c1c194299fd412ab1d7529a9e43a456. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:24.419633 11977 raft_consensus.cc:348] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:24.419780 11977 raft_consensus.cc:374] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:24.419823 11977 raft_consensus.cc:729] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a25f19d5dc774109978ca0c1e7a3609a, State: Initialized, Role: FOLLOWER
I20260806 17:02:24.419961 11977 consensus_queue.cc:260] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:24.420019 12027 tablet_replica.cc:330] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a: stopping tablet replica
I20260806 17:02:24.420500 12027 raft_consensus.cc:2229] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:24.420818 11977 ts_tablet_manager.cc:1428] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 17:02:24.421236 12027 raft_consensus.cc:2258] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:24.422864 12027 ts_tablet_manager.cc:1905] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:24.425688 12027 ts_tablet_manager.cc:1918] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 17:02:24.425902 12027 log.cc:1196] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/fc058498ee8f432c94ecd2caadca6120
I20260806 17:02:24.426319 12027 ts_tablet_manager.cc:1939] T fc058498ee8f432c94ecd2caadca6120 P a25f19d5dc774109978ca0c1e7a3609a: Deleting consensus metadata
I20260806 17:02:24.426960 11439 catalog_manager.cc:4841] TS e114e7ba5e8f4ff08f027a7ca2a3d25e (127.9.111.66:44123): tablet fc058498ee8f432c94ecd2caadca6120 (table test-table [id=8775289c8c2d4c8da053122ce3c371fc]) successfully deleted
I20260806 17:02:24.427016 11975 tablet_bootstrap.cc:492] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Bootstrap starting.
I20260806 17:02:24.427804 11438 catalog_manager.cc:4841] TS a25f19d5dc774109978ca0c1e7a3609a (127.9.111.67:44043): tablet fc058498ee8f432c94ecd2caadca6120 (table test-table [id=8775289c8c2d4c8da053122ce3c371fc]) successfully deleted
I20260806 17:02:24.427919 11975 tablet_bootstrap.cc:654] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:24.429247 11975 tablet_bootstrap.cc:492] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e: No bootstrap required, opened a new log
I20260806 17:02:24.429342 11975 ts_tablet_manager.cc:1397] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 17:02:24.429751 11975 raft_consensus.cc:348] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:24.429939 11975 raft_consensus.cc:374] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:24.429975 11975 raft_consensus.cc:729] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e114e7ba5e8f4ff08f027a7ca2a3d25e, State: Initialized, Role: FOLLOWER
I20260806 17:02:24.430117 11975 consensus_queue.cc:260] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:24.430387 11975 ts_tablet_manager.cc:1428] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 17:02:24.532795 11983 raft_consensus.cc:482] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:24.532956 11983 raft_consensus.cc:504] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:24.534173 11983 leader_election.cc:288] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5d8a42e20494b5d8905c8dc0222ad74 (127.9.111.65:41463), e114e7ba5e8f4ff08f027a7ca2a3d25e (127.9.111.66:44123)
I20260806 17:02:24.539125 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.534425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:41463 (local address 127.9.111.67:47327)
0806 17:02:24.534989 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:24.535004 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:24.535136 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:24.535516 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:24.535519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:24.535538 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:24.535659 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:24.535666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.536669 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.536672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:24.537752 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:24.537762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.538599 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.538603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:24.538727 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.538756 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 17:02:24.538855 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 17:02:24.538938 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":313,"thread_start_us":113,"threads_started":1}
I20260806 17:02:24.539121 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.534623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:47327 (local address 127.9.111.65:41463)
0806 17:02:24.534793 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:24.534798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:24.535066 (+ 268us) server_negotiation.cc:407] Connection header received
0806 17:02:24.535303 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:24.535306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:24.535386 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:24.535461 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:24.535797 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.536520 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.537930 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.538452 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.538579 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.538645 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:24.538898 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 17:02:24.538982 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 17:02:24.539590 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.534578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:56031 (local address 127.9.111.66:44123)
0806 17:02:24.534792 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:24.534798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:24.535044 (+ 246us) server_negotiation.cc:407] Connection header received
0806 17:02:24.535203 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:24.535207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:24.535269 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:24.535367 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:24.535701 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.536501 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.538272 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.538810 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.538964 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.539013 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:24.539370 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 17:02:24.539475 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260806 17:02:24.539798 11617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c1c194299fd412ab1d7529a9e43a456" candidate_uuid: "a25f19d5dc774109978ca0c1e7a3609a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" is_pre_election: true
I20260806 17:02:24.539973 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.534425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:44123 (local address 127.9.111.67:56031)
0806 17:02:24.534914 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:24.534930 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:24.535078 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:24.535391 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:24.535393 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:24.535414 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:24.535564 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:24.535571 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.536642 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.536646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:24.538120 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:24.538132 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.539073 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.539076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:24.539197 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.539230 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 17:02:24.539730 (+ 500us) client_negotiation.cc:240] Negotiation successful
0806 17:02:24.539808 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":34,"thread_start_us":75,"threads_started":1}
I20260806 17:02:24.539974 11617 raft_consensus.cc:2454] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a25f19d5dc774109978ca0c1e7a3609a in term 0.
I20260806 17:02:24.540431 11761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c1c194299fd412ab1d7529a9e43a456" candidate_uuid: "a25f19d5dc774109978ca0c1e7a3609a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" is_pre_election: true
I20260806 17:02:24.540436 11836 leader_election.cc:302] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a25f19d5dc774109978ca0c1e7a3609a, c5d8a42e20494b5d8905c8dc0222ad74; no voters:
I20260806 17:02:24.540589 11761 raft_consensus.cc:2454] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a25f19d5dc774109978ca0c1e7a3609a in term 0.
I20260806 17:02:24.540812 11983 raft_consensus.cc:2781] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:02:24.540881 11983 raft_consensus.cc:482] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:02:24.540930 11983 raft_consensus.cc:3037] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:24.542061 11983 raft_consensus.cc:504] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:24.542496 11983 leader_election.cc:288] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [CANDIDATE]: Term 1 election: Requested vote from peers c5d8a42e20494b5d8905c8dc0222ad74 (127.9.111.65:41463), e114e7ba5e8f4ff08f027a7ca2a3d25e (127.9.111.66:44123)
I20260806 17:02:24.542903 11617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c1c194299fd412ab1d7529a9e43a456" candidate_uuid: "a25f19d5dc774109978ca0c1e7a3609a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5d8a42e20494b5d8905c8dc0222ad74"
I20260806 17:02:24.542969 11761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c1c194299fd412ab1d7529a9e43a456" candidate_uuid: "a25f19d5dc774109978ca0c1e7a3609a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e"
I20260806 17:02:24.543051 11617 raft_consensus.cc:3037] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:24.543082 11761 raft_consensus.cc:3037] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:24.544628 11617 raft_consensus.cc:2454] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a25f19d5dc774109978ca0c1e7a3609a in term 1.
I20260806 17:02:24.544622 11761 raft_consensus.cc:2454] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a25f19d5dc774109978ca0c1e7a3609a in term 1.
I20260806 17:02:24.545054 11836 leader_election.cc:302] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a25f19d5dc774109978ca0c1e7a3609a, c5d8a42e20494b5d8905c8dc0222ad74; no voters:
I20260806 17:02:24.545281 11983 raft_consensus.cc:2781] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:24.545588 11983 raft_consensus.cc:686] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [term 1 LEADER]: Becoming Leader. State: Replica: a25f19d5dc774109978ca0c1e7a3609a, State: Running, Role: LEADER
I20260806 17:02:24.545930 11983 consensus_queue.cc:237] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:24.549316 11453 catalog_manager.cc:5495] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a reported cstate change: term changed from 0 to 1, leader changed from <none> to a25f19d5dc774109978ca0c1e7a3609a (127.9.111.67). New cstate: current_term: 1 leader_uuid: "a25f19d5dc774109978ca0c1e7a3609a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } health_report { overall_health: HEALTHY } } }
I20260806 17:02:24.597473 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.593251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.9.111.67:44043)
0806 17:02:24.593464 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:24.593469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:24.593486 (+ 17us) server_negotiation.cc:407] Connection header received
0806 17:02:24.593656 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:24.593658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:24.593721 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:24.593966 (+ 245us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:24.594240 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.594887 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.595810 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.596076 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.596199 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.596518 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:24.597012 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:24.597152 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:24.597225 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 17:02:24.597293 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260806 17:02:24.609619 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.605798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:58345 (local address 127.9.111.66:44123)
0806 17:02:24.605969 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:24.605974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:24.606124 (+ 150us) server_negotiation.cc:407] Connection header received
0806 17:02:24.606289 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:24.606292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:24.606348 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:24.606435 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:24.607040 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.607599 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.608468 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.608709 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.608826 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.609125 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:24.609205 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:24.609289 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:24.609409 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 17:02:24.609467 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 17:02:24.609647 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.605597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:44123 (local address 127.9.111.67:58345)
0806 17:02:24.606028 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:24.606041 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:24.606179 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:24.606448 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:24.606451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:24.606460 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 17:02:24.606918 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:24.606925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.607720 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.607724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:24.608337 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:24.608345 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.608830 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.608833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:24.608928 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.608974 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 17:02:24.609295 (+ 321us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 17:02:24.609304 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 17:02:24.609386 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 17:02:24.609473 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":159,"threads_started":1}
I20260806 17:02:24.617981 11761 raft_consensus.cc:1261] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 FOLLOWER]: Refusing update from remote peer a25f19d5dc774109978ca0c1e7a3609a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:24.618052 11617 raft_consensus.cc:1261] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74 [term 1 FOLLOWER]: Refusing update from remote peer a25f19d5dc774109978ca0c1e7a3609a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:24.618696 12034 consensus_queue.cc:1035] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:24.618927 11983 consensus_queue.cc:1035] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:24.641764 12022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.638177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:44043 (local address 127.9.111.66:39927)
0806 17:02:24.638326 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:24.638344 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:24.638431 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:24.638812 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:24.638816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:24.638824 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 17:02:24.638941 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:24.638949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.639977 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.639980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:24.640614 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:24.640624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.641079 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.641082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:24.641150 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.641178 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 17:02:24.641462 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 17:02:24.641468 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 17:02:24.641538 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 17:02:24.641625 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260806 17:02:24.641763 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:24.638245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:39927 (local address 127.9.111.67:44043)
0806 17:02:24.638356 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:24.638364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:24.638396 (+ 32us) server_negotiation.cc:407] Connection header received
0806 17:02:24.638551 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:24.638555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:24.638611 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:24.638683 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:24.639095 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.639858 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.640753 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:24.640975 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:24.641060 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:24.641283 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:24.641380 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:24.641434 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:24.641557 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 17:02:24.641603 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260806 17:02:24.642771 12006 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 17:02:26.207581 11454 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:26.207782 11454 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:26.210469 11454 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024: Sending DeleteTablet for 3 replicas of tablet 4c1c194299fd412ab1d7529a9e43a456
I20260806 17:02:26.211077 11597 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c1c194299fd412ab1d7529a9e43a456 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:26 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:49464
I20260806 17:02:26.211086 11741 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c1c194299fd412ab1d7529a9e43a456 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:26 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:36714
I20260806 17:02:26.211198 11885 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c1c194299fd412ab1d7529a9e43a456 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:26 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:37764
I20260806 17:02:26.211634 12066 tablet_replica.cc:330] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e: stopping tablet replica
I20260806 17:02:26.211746 12069 tablet_replica.cc:330] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a: stopping tablet replica
I20260806 17:02:26.211854 12066 raft_consensus.cc:2229] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:26.211901 12069 raft_consensus.cc:2229] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:02:26.212811 12066 raft_consensus.cc:2258] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:26.213541 12066 ts_tablet_manager.cc:1905] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:26.215360 12069 raft_consensus.cc:2258] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:26.216033 12069 ts_tablet_manager.cc:1905] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:26.216276 12068 tablet_replica.cc:330] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74: stopping tablet replica
I20260806 17:02:26.216452 12068 raft_consensus.cc:2229] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:26.216610 12068 raft_consensus.cc:2258] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:26.217178 12066 ts_tablet_manager.cc:1918] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 17:02:26.217314 12066 log.cc:1196] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/4c1c194299fd412ab1d7529a9e43a456
I20260806 17:02:26.217324 12068 ts_tablet_manager.cc:1905] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:26.217705 12066 ts_tablet_manager.cc:1939] T 4c1c194299fd412ab1d7529a9e43a456 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Deleting consensus metadata
I20260806 17:02:26.218655 11439 catalog_manager.cc:4841] TS e114e7ba5e8f4ff08f027a7ca2a3d25e (127.9.111.66:44123): tablet 4c1c194299fd412ab1d7529a9e43a456 (table test-table [id=028b9b3a1b154089aa28795286c66b99]) successfully deleted
I20260806 17:02:26.218863 12069 ts_tablet_manager.cc:1918] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 17:02:26.219017 12069 log.cc:1196] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/4c1c194299fd412ab1d7529a9e43a456
I20260806 17:02:26.219329 12069 ts_tablet_manager.cc:1939] T 4c1c194299fd412ab1d7529a9e43a456 P a25f19d5dc774109978ca0c1e7a3609a: Deleting consensus metadata
I20260806 17:02:26.219928 12068 ts_tablet_manager.cc:1918] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 17:02:26.220059 12068 log.cc:1196] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/wal/wals/4c1c194299fd412ab1d7529a9e43a456
I20260806 17:02:26.220013 11438 catalog_manager.cc:4841] TS a25f19d5dc774109978ca0c1e7a3609a (127.9.111.67:44043): tablet 4c1c194299fd412ab1d7529a9e43a456 (table test-table [id=028b9b3a1b154089aa28795286c66b99]) successfully deleted
I20260806 17:02:26.220366 12068 ts_tablet_manager.cc:1939] T 4c1c194299fd412ab1d7529a9e43a456 P c5d8a42e20494b5d8905c8dc0222ad74: Deleting consensus metadata
I20260806 17:02:26.221031 11437 catalog_manager.cc:4841] TS c5d8a42e20494b5d8905c8dc0222ad74 (127.9.111.65:41463): tablet 4c1c194299fd412ab1d7529a9e43a456 (table test-table [id=028b9b3a1b154089aa28795286c66b99]) successfully deleted
Aug 06 17:02:26 dist-test-slave-69rg krb5kdc[11406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035746, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 17:02:26 dist-test-slave-69rg krb5kdc[11406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035746, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:26.247925 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:26.238709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54006 (local address 127.9.111.126:43945)
0806 17:02:26.239009 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:26.239017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:26.239033 (+ 16us) server_negotiation.cc:407] Connection header received
0806 17:02:26.239080 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:26.239083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:26.239125 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:26.239215 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:26.240031 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:26.240598 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:26.241754 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:26.242038 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:26.242125 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:26.244794 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:26.244797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:26.244799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:26.244826 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:26.246340 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:26.246766 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:26.246770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:26.246771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:26.246859 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:26.247112 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:26.247114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:26.247115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:26.247306 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:26.247413 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:26.247669 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 17:02:26.247787 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":138,"threads_started":1}
I20260806 17:02:26.250725 11454 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:02:26.251176 11454 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:02:26.256181 11597 tablet_service.cc:1467] Processing CreateTablet for tablet a1273dfa06914ed880b1acbe3049b7b1 (DEFAULT_TABLE table=test-table [id=5a7130e97b2340738447e9bbd0db9a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:26.256399 11885 tablet_service.cc:1467] Processing CreateTablet for tablet a1273dfa06914ed880b1acbe3049b7b1 (DEFAULT_TABLE table=test-table [id=5a7130e97b2340738447e9bbd0db9a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:26.256564 11597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1273dfa06914ed880b1acbe3049b7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:26.256704 11885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1273dfa06914ed880b1acbe3049b7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:26.258858 12078 tablet_bootstrap.cc:492] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74: Bootstrap starting.
I20260806 17:02:26.258931 12079 tablet_bootstrap.cc:492] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a: Bootstrap starting.
I20260806 17:02:26.259085 11741 tablet_service.cc:1467] Processing CreateTablet for tablet a1273dfa06914ed880b1acbe3049b7b1 (DEFAULT_TABLE table=test-table [id=5a7130e97b2340738447e9bbd0db9a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:26.259363 11741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1273dfa06914ed880b1acbe3049b7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:26.259824 12078 tablet_bootstrap.cc:654] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:26.259959 12079 tablet_bootstrap.cc:654] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:26.261071 12078 tablet_bootstrap.cc:492] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74: No bootstrap required, opened a new log
I20260806 17:02:26.261183 12078 ts_tablet_manager.cc:1397] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 17:02:26.261268 12079 tablet_bootstrap.cc:492] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a: No bootstrap required, opened a new log
I20260806 17:02:26.261360 12079 ts_tablet_manager.cc:1397] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 17:02:26.261628 12078 raft_consensus.cc:348] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:26.261758 12078 raft_consensus.cc:374] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:26.261802 12078 raft_consensus.cc:729] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d8a42e20494b5d8905c8dc0222ad74, State: Initialized, Role: FOLLOWER
I20260806 17:02:26.261790 12079 raft_consensus.cc:348] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:26.261924 12079 raft_consensus.cc:374] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:26.261968 12079 raft_consensus.cc:729] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a25f19d5dc774109978ca0c1e7a3609a, State: Initialized, Role: FOLLOWER
I20260806 17:02:26.261976 12078 consensus_queue.cc:260] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:26.262148 12082 tablet_bootstrap.cc:492] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Bootstrap starting.
I20260806 17:02:26.262133 12079 consensus_queue.cc:260] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:26.262385 12078 ts_tablet_manager.cc:1428] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 17:02:26.262454 12079 ts_tablet_manager.cc:1428] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 17:02:26.264273 12082 tablet_bootstrap.cc:654] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:26.266364 12082 tablet_bootstrap.cc:492] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e: No bootstrap required, opened a new log
I20260806 17:02:26.266487 12082 ts_tablet_manager.cc:1397] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 17:02:26.266928 12082 raft_consensus.cc:348] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:26.267026 12082 raft_consensus.cc:374] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:26.267076 12082 raft_consensus.cc:729] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e114e7ba5e8f4ff08f027a7ca2a3d25e, State: Initialized, Role: FOLLOWER
I20260806 17:02:26.267205 12082 consensus_queue.cc:260] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:26.267493 12082 ts_tablet_manager.cc:1428] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 17:02:26.384444 12046 raft_consensus.cc:482] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:26.384588 12046 raft_consensus.cc:504] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:26.385098 12046 leader_election.cc:288] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5d8a42e20494b5d8905c8dc0222ad74 (127.9.111.65:41463), a25f19d5dc774109978ca0c1e7a3609a (127.9.111.67:44043)
I20260806 17:02:26.385612 11905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1273dfa06914ed880b1acbe3049b7b1" candidate_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a25f19d5dc774109978ca0c1e7a3609a" is_pre_election: true
I20260806 17:02:26.385663 11617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1273dfa06914ed880b1acbe3049b7b1" candidate_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" is_pre_election: true
I20260806 17:02:26.385788 11905 raft_consensus.cc:2454] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e114e7ba5e8f4ff08f027a7ca2a3d25e in term 0.
I20260806 17:02:26.385865 11617 raft_consensus.cc:2454] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e114e7ba5e8f4ff08f027a7ca2a3d25e in term 0.
I20260806 17:02:26.386202 11693 leader_election.cc:302] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a25f19d5dc774109978ca0c1e7a3609a, e114e7ba5e8f4ff08f027a7ca2a3d25e; no voters:
I20260806 17:02:26.386432 12046 raft_consensus.cc:2781] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:02:26.386509 12046 raft_consensus.cc:482] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:02:26.386570 12046 raft_consensus.cc:3037] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:26.387666 12046 raft_consensus.cc:504] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:26.388043 12046 leader_election.cc:288] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [CANDIDATE]: Term 1 election: Requested vote from peers c5d8a42e20494b5d8905c8dc0222ad74 (127.9.111.65:41463), a25f19d5dc774109978ca0c1e7a3609a (127.9.111.67:44043)
I20260806 17:02:26.388461 11617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1273dfa06914ed880b1acbe3049b7b1" candidate_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5d8a42e20494b5d8905c8dc0222ad74"
I20260806 17:02:26.388507 11905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1273dfa06914ed880b1acbe3049b7b1" candidate_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a25f19d5dc774109978ca0c1e7a3609a"
I20260806 17:02:26.388582 11617 raft_consensus.cc:3037] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:26.388679 11905 raft_consensus.cc:3037] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:26.389643 11617 raft_consensus.cc:2454] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e114e7ba5e8f4ff08f027a7ca2a3d25e in term 1.
I20260806 17:02:26.389654 11905 raft_consensus.cc:2454] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e114e7ba5e8f4ff08f027a7ca2a3d25e in term 1.
I20260806 17:02:26.390094 11693 leader_election.cc:302] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a25f19d5dc774109978ca0c1e7a3609a, e114e7ba5e8f4ff08f027a7ca2a3d25e; no voters:
I20260806 17:02:26.390754 12046 raft_consensus.cc:2781] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:26.390846 12046 raft_consensus.cc:686] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 LEADER]: Becoming Leader. State: Replica: e114e7ba5e8f4ff08f027a7ca2a3d25e, State: Running, Role: LEADER
I20260806 17:02:26.390998 12046 consensus_queue.cc:237] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:26.392928 11453 catalog_manager.cc:5495] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e reported cstate change: term changed from 0 to 1, leader changed from <none> to e114e7ba5e8f4ff08f027a7ca2a3d25e (127.9.111.66). New cstate: current_term: 1 leader_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } health_report { overall_health: UNKNOWN } } }
I20260806 17:02:26.425562 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:26.422509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36744 (local address 127.9.111.66:44123)
0806 17:02:26.422799 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:26.422807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:26.422819 (+ 12us) server_negotiation.cc:407] Connection header received
0806 17:02:26.422857 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:26.422859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:26.422900 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:26.422987 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:26.423259 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:26.423810 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:26.424623 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:26.424844 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:26.424918 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:26.425177 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:26.425242 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:26.425297 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:26.425403 (+ 106us) server_negotiation.cc:299] Negotiation successful
0806 17:02:26.425444 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":172,"threads_started":1}
I20260806 17:02:26.435981 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:26.432919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:58141 (local address 127.9.111.66:44123)
0806 17:02:26.432995 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:26.433001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:26.433049 (+ 48us) server_negotiation.cc:407] Connection header received
0806 17:02:26.433239 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:26.433241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:26.433277 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:26.433351 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:26.433607 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:26.434166 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:26.434964 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:26.435176 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:26.435280 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:26.435513 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:26.435571 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:26.435649 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:26.435732 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 17:02:26.435827 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":27}
I20260806 17:02:26.435981 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:26.432725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:44123 (local address 127.9.111.66:58141)
0806 17:02:26.432974 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:26.432990 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:26.433095 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:26.433368 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:26.433370 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:26.433376 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 17:02:26.433505 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:26.433516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:26.434264 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:26.434271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:26.434854 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:26.434861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:26.435280 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:26.435283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:26.435377 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:26.435398 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 17:02:26.435654 (+ 256us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 17:02:26.435662 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 17:02:26.435724 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 17:02:26.435851 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":53,"thread_start_us":108,"threads_started":1}
I20260806 17:02:26.441500 11617 raft_consensus.cc:1261] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74 [term 1 FOLLOWER]: Refusing update from remote peer e114e7ba5e8f4ff08f027a7ca2a3d25e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:26.441553 11905 raft_consensus.cc:1261] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a [term 1 FOLLOWER]: Refusing update from remote peer e114e7ba5e8f4ff08f027a7ca2a3d25e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:26.441978 12051 consensus_queue.cc:1035] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:26.442124 12036 consensus_queue.cc:1035] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:26.476723 11454 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:26.476882 11454 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:26.479261 11454 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024: Sending DeleteTablet for 3 replicas of tablet a1273dfa06914ed880b1acbe3049b7b1
I20260806 17:02:26.479851 11597 tablet_service.cc:1514] Processing DeleteTablet for tablet a1273dfa06914ed880b1acbe3049b7b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:26 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:49464
I20260806 17:02:26.479874 11741 tablet_service.cc:1514] Processing DeleteTablet for tablet a1273dfa06914ed880b1acbe3049b7b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:26 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:36714
I20260806 17:02:26.479902 11885 tablet_service.cc:1514] Processing DeleteTablet for tablet a1273dfa06914ed880b1acbe3049b7b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:26 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:37764
I20260806 17:02:26.480048 12068 tablet_replica.cc:330] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74: stopping tablet replica
I20260806 17:02:26.480073 12066 tablet_replica.cc:330] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e: stopping tablet replica
I20260806 17:02:26.480165 12066 raft_consensus.cc:2229] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:02:26.480166 12068 raft_consensus.cc:2229] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:26.480298 12068 raft_consensus.cc:2258] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:26.480439 12066 raft_consensus.cc:2258] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:26.480656 12069 tablet_replica.cc:330] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a: stopping tablet replica
I20260806 17:02:26.480768 12069 raft_consensus.cc:2229] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:26.480885 12069 raft_consensus.cc:2258] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:26.481163 12068 ts_tablet_manager.cc:1905] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:26.481588 12066 ts_tablet_manager.cc:1905] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:26.482314 11453 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 17:02:26.482455 12069 ts_tablet_manager.cc:1905] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 17:02:26.482718 11453 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:02:26.484159 12066 ts_tablet_manager.cc:1918] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 17:02:26.484212 12066 log.cc:1196] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/a1273dfa06914ed880b1acbe3049b7b1
I20260806 17:02:26.484449 12066 ts_tablet_manager.cc:1939] T a1273dfa06914ed880b1acbe3049b7b1 P e114e7ba5e8f4ff08f027a7ca2a3d25e: Deleting consensus metadata
I20260806 17:02:26.484910 12069 ts_tablet_manager.cc:1918] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 17:02:26.484974 12069 log.cc:1196] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/a1273dfa06914ed880b1acbe3049b7b1
I20260806 17:02:26.485149 11439 catalog_manager.cc:4841] TS e114e7ba5e8f4ff08f027a7ca2a3d25e (127.9.111.66:44123): tablet a1273dfa06914ed880b1acbe3049b7b1 (table test-table [id=5a7130e97b2340738447e9bbd0db9a81]) successfully deleted
I20260806 17:02:26.485215 12069 ts_tablet_manager.cc:1939] T a1273dfa06914ed880b1acbe3049b7b1 P a25f19d5dc774109978ca0c1e7a3609a: Deleting consensus metadata
I20260806 17:02:26.485839 11438 catalog_manager.cc:4841] TS a25f19d5dc774109978ca0c1e7a3609a (127.9.111.67:44043): tablet a1273dfa06914ed880b1acbe3049b7b1 (table test-table [id=5a7130e97b2340738447e9bbd0db9a81]) successfully deleted
I20260806 17:02:26.488811 12068 ts_tablet_manager.cc:1918] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 17:02:26.488909 12068 log.cc:1196] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-0/wal/wals/a1273dfa06914ed880b1acbe3049b7b1
I20260806 17:02:26.489264 12068 ts_tablet_manager.cc:1939] T a1273dfa06914ed880b1acbe3049b7b1 P c5d8a42e20494b5d8905c8dc0222ad74: Deleting consensus metadata
I20260806 17:02:26.489924 11437 catalog_manager.cc:4841] TS c5d8a42e20494b5d8905c8dc0222ad74 (127.9.111.65:41463): tablet a1273dfa06914ed880b1acbe3049b7b1 (table test-table [id=5a7130e97b2340738447e9bbd0db9a81]) successfully deleted
I20260806 17:02:26.490263 11597 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9b7d525ad4443e94c09966618fcbee (DEFAULT_TABLE table=test-table [id=6047c635aba94e0aa2375e24e25a2b13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:26.490535 11597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9b7d525ad4443e94c09966618fcbee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:26.492208 12078 tablet_bootstrap.cc:492] T 9d9b7d525ad4443e94c09966618fcbee P c5d8a42e20494b5d8905c8dc0222ad74: Bootstrap starting.
I20260806 17:02:26.492420 11885 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9b7d525ad4443e94c09966618fcbee (DEFAULT_TABLE table=test-table [id=6047c635aba94e0aa2375e24e25a2b13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:26.492671 11885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9b7d525ad4443e94c09966618fcbee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:26.493017 12078 tablet_bootstrap.cc:654] T 9d9b7d525ad4443e94c09966618fcbee P c5d8a42e20494b5d8905c8dc0222ad74: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:26.494315 12078 tablet_bootstrap.cc:492] T 9d9b7d525ad4443e94c09966618fcbee P c5d8a42e20494b5d8905c8dc0222ad74: No bootstrap required, opened a new log
I20260806 17:02:26.494390 12078 ts_tablet_manager.cc:1397] T 9d9b7d525ad4443e94c09966618fcbee P c5d8a42e20494b5d8905c8dc0222ad74: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 17:02:26.494776 12079 tablet_bootstrap.cc:492] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a: Bootstrap starting.
I20260806 17:02:26.494781 12078 raft_consensus.cc:348] T 9d9b7d525ad4443e94c09966618fcbee P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:26.494927 12078 raft_consensus.cc:374] T 9d9b7d525ad4443e94c09966618fcbee P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:26.494961 12078 raft_consensus.cc:729] T 9d9b7d525ad4443e94c09966618fcbee P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d8a42e20494b5d8905c8dc0222ad74, State: Initialized, Role: FOLLOWER
I20260806 17:02:26.495083 12078 consensus_queue.cc:260] T 9d9b7d525ad4443e94c09966618fcbee P c5d8a42e20494b5d8905c8dc0222ad74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:26.495330 12078 ts_tablet_manager.cc:1428] T 9d9b7d525ad4443e94c09966618fcbee P c5d8a42e20494b5d8905c8dc0222ad74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 17:02:26.495456 11741 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9b7d525ad4443e94c09966618fcbee (DEFAULT_TABLE table=test-table [id=6047c635aba94e0aa2375e24e25a2b13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:26.495673 12079 tablet_bootstrap.cc:654] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:26.495723 11741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9b7d525ad4443e94c09966618fcbee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:26.498279 12082 tablet_bootstrap.cc:492] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e: Bootstrap starting.
I20260806 17:02:26.498726 12079 tablet_bootstrap.cc:492] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a: No bootstrap required, opened a new log
I20260806 17:02:26.498847 12079 ts_tablet_manager.cc:1397] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260806 17:02:26.499105 12082 tablet_bootstrap.cc:654] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:26.499297 12079 raft_consensus.cc:348] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:26.499419 12079 raft_consensus.cc:374] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:26.499452 12079 raft_consensus.cc:729] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a25f19d5dc774109978ca0c1e7a3609a, State: Initialized, Role: FOLLOWER
I20260806 17:02:26.499614 12079 consensus_queue.cc:260] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:26.499894 12079 ts_tablet_manager.cc:1428] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 17:02:26.500197 12082 tablet_bootstrap.cc:492] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e: No bootstrap required, opened a new log
I20260806 17:02:26.500279 12082 ts_tablet_manager.cc:1397] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 17:02:26.500630 12082 raft_consensus.cc:348] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:26.500736 12082 raft_consensus.cc:374] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:26.500766 12082 raft_consensus.cc:729] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e114e7ba5e8f4ff08f027a7ca2a3d25e, State: Initialized, Role: FOLLOWER
I20260806 17:02:26.500885 12082 consensus_queue.cc:260] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:26.501116 12082 ts_tablet_manager.cc:1428] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 17:02:26.599766 12051 raft_consensus.cc:482] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:26.599975 12051 raft_consensus.cc:504] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:26.600449 12051 leader_election.cc:288] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5d8a42e20494b5d8905c8dc0222ad74 (127.9.111.65:41463), a25f19d5dc774109978ca0c1e7a3609a (127.9.111.67:44043)
I20260806 17:02:26.600966 11617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9b7d525ad4443e94c09966618fcbee" candidate_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" is_pre_election: true
I20260806 17:02:26.600976 11904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9b7d525ad4443e94c09966618fcbee" candidate_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a25f19d5dc774109978ca0c1e7a3609a" is_pre_election: true
I20260806 17:02:26.601126 11617 raft_consensus.cc:2454] T 9d9b7d525ad4443e94c09966618fcbee P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e114e7ba5e8f4ff08f027a7ca2a3d25e in term 0.
I20260806 17:02:26.601154 11904 raft_consensus.cc:2454] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e114e7ba5e8f4ff08f027a7ca2a3d25e in term 0.
I20260806 17:02:26.601465 11692 leader_election.cc:302] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5d8a42e20494b5d8905c8dc0222ad74, e114e7ba5e8f4ff08f027a7ca2a3d25e; no voters:
I20260806 17:02:26.601667 12051 raft_consensus.cc:2781] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:02:26.601728 12051 raft_consensus.cc:482] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:02:26.601787 12051 raft_consensus.cc:3037] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:26.603008 12051 raft_consensus.cc:504] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:26.603411 12051 leader_election.cc:288] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [CANDIDATE]: Term 1 election: Requested vote from peers c5d8a42e20494b5d8905c8dc0222ad74 (127.9.111.65:41463), a25f19d5dc774109978ca0c1e7a3609a (127.9.111.67:44043)
I20260806 17:02:26.603794 11617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9b7d525ad4443e94c09966618fcbee" candidate_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5d8a42e20494b5d8905c8dc0222ad74"
I20260806 17:02:26.603832 11904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9b7d525ad4443e94c09966618fcbee" candidate_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a25f19d5dc774109978ca0c1e7a3609a"
I20260806 17:02:26.603956 11904 raft_consensus.cc:3037] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:26.603956 11617 raft_consensus.cc:3037] T 9d9b7d525ad4443e94c09966618fcbee P c5d8a42e20494b5d8905c8dc0222ad74 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:26.604892 11617 raft_consensus.cc:2454] T 9d9b7d525ad4443e94c09966618fcbee P c5d8a42e20494b5d8905c8dc0222ad74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e114e7ba5e8f4ff08f027a7ca2a3d25e in term 1.
I20260806 17:02:26.605206 11692 leader_election.cc:302] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5d8a42e20494b5d8905c8dc0222ad74, e114e7ba5e8f4ff08f027a7ca2a3d25e; no voters:
I20260806 17:02:26.605345 11904 raft_consensus.cc:2454] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e114e7ba5e8f4ff08f027a7ca2a3d25e in term 1.
I20260806 17:02:26.605408 12051 raft_consensus.cc:2781] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:26.605522 12051 raft_consensus.cc:686] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 LEADER]: Becoming Leader. State: Replica: e114e7ba5e8f4ff08f027a7ca2a3d25e, State: Running, Role: LEADER
I20260806 17:02:26.605685 12051 consensus_queue.cc:237] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } }
I20260806 17:02:26.607611 11453 catalog_manager.cc:5495] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e reported cstate change: term changed from 0 to 1, leader changed from <none> to e114e7ba5e8f4ff08f027a7ca2a3d25e (127.9.111.66). New cstate: current_term: 1 leader_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e114e7ba5e8f4ff08f027a7ca2a3d25e" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 } health_report { overall_health: UNKNOWN } } }
I20260806 17:02:26.631430 11904 raft_consensus.cc:1261] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a [term 1 FOLLOWER]: Refusing update from remote peer e114e7ba5e8f4ff08f027a7ca2a3d25e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:26.631431 11618 raft_consensus.cc:1261] T 9d9b7d525ad4443e94c09966618fcbee P c5d8a42e20494b5d8905c8dc0222ad74 [term 1 FOLLOWER]: Refusing update from remote peer e114e7ba5e8f4ff08f027a7ca2a3d25e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:26.631870 12051 consensus_queue.cc:1035] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5d8a42e20494b5d8905c8dc0222ad74" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:26.632071 12036 consensus_queue.cc:1035] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a25f19d5dc774109978ca0c1e7a3609a" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 44043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 17:02:26.646143 12044 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 17:02:28.200664 11454 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:28.200840 11454 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:28.203621 11454 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5887e2cf64941ae8b43e9a19d75d024: Sending DeleteTablet for 3 replicas of tablet 9d9b7d525ad4443e94c09966618fcbee
I20260806 17:02:28.204288 11597 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d9b7d525ad4443e94c09966618fcbee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:28 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:49464
I20260806 17:02:28.204435 11741 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d9b7d525ad4443e94c09966618fcbee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:28 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:36714
I20260806 17:02:28.204455 11885 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d9b7d525ad4443e94c09966618fcbee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:28 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:37764
I20260806 17:02:28.204762 12122 tablet_replica.cc:330] T 9d9b7d525ad4443e94c09966618fcbee P c5d8a42e20494b5d8905c8dc0222ad74: stopping tablet replica
I20260806 17:02:28.204815 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 11528
I20260806 17:02:28.206328 12123 tablet_replica.cc:330] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e: stopping tablet replica
I20260806 17:02:28.206514 12123 raft_consensus.cc:2229] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:02:28.206707 12123 raft_consensus.cc:2258] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:28.207022 12124 tablet_replica.cc:330] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a: stopping tablet replica
I20260806 17:02:28.207196 12124 raft_consensus.cc:2229] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:28.207315 12124 raft_consensus.cc:2258] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:28.207768 12123 ts_tablet_manager.cc:1905] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:28.210291 12124 ts_tablet_manager.cc:1905] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:28.211410 12123 ts_tablet_manager.cc:1918] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 17:02:28.211508 12123 log.cc:1196] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/9d9b7d525ad4443e94c09966618fcbee
I20260806 17:02:28.211829 12123 ts_tablet_manager.cc:1939] T 9d9b7d525ad4443e94c09966618fcbee P e114e7ba5e8f4ff08f027a7ca2a3d25e: Deleting consensus metadata
I20260806 17:02:28.212682 12124 ts_tablet_manager.cc:1918] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 17:02:28.212800 12124 log.cc:1196] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/9d9b7d525ad4443e94c09966618fcbee
I20260806 17:02:28.212769 11439 catalog_manager.cc:4841] TS e114e7ba5e8f4ff08f027a7ca2a3d25e (127.9.111.66:44123): tablet 9d9b7d525ad4443e94c09966618fcbee (table test-table [id=6047c635aba94e0aa2375e24e25a2b13]) successfully deleted
I20260806 17:02:28.213135 12124 ts_tablet_manager.cc:1939] T 9d9b7d525ad4443e94c09966618fcbee P a25f19d5dc774109978ca0c1e7a3609a: Deleting consensus metadata
I20260806 17:02:28.213984 11438 catalog_manager.cc:4841] TS a25f19d5dc774109978ca0c1e7a3609a (127.9.111.67:44043): tablet 9d9b7d525ad4443e94c09966618fcbee (table test-table [id=6047c635aba94e0aa2375e24e25a2b13]) successfully deleted
W20260806 17:02:28.215598 11437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.111.65:41463 (error 108)
I20260806 17:02:28.216612 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 11674
I20260806 17:02:28.218055 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:28.216465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:41463 (local address 127.0.0.1:49484)
0806 17:02:28.217720 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:28.217906 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.65:41463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":80,"threads_started":1}
W20260806 17:02:28.218374 11437 catalog_manager.cc:4584] TS c5d8a42e20494b5d8905c8dc0222ad74 (127.9.111.65:41463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d9b7d525ad4443e94c09966618fcbee: Network error: Client connection negotiation failed: client connection to 127.9.111.65:41463: connect: Connection refused (error 111)
I20260806 17:02:28.225530 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 11819
I20260806 17:02:28.233593 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 11420
2026-08-06T17:02:28Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6765 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:02:28 dist-test-slave-69rg krb5kdc[12129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:02:28 dist-test-slave-69rg krb5kdc[12129](info): set up 2 sockets
Aug 06 17:02:28 dist-test-slave-69rg krb5kdc[12129](info): commencing operation
krb5kdc: starting...
W20260806 17:02:29.321204 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:02:29 dist-test-slave-69rg krb5kdc[12129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:02:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:02:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:02:29.459465 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:45633
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:45633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:29.573625 12142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:29.573976 12142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:29.574020 12142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:29.578881 12142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:29.578950 12142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:29.578971 12142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:29.578989 12142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:29.579005 12142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:02:29.584606 12142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:45633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:45633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:29.585773 12142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:29.586859 12142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 17:02:29.596076 12142 server_base.cc:1034] running on GCE node
W20260806 17:02:29.596012 12150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:29.596006 12148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:29.596067 12147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:29.596987 12142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:29.598284 12142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:29.599494 12142 hybrid_clock.cc:648] HybridClock initialized: now 1786035749599464 us; error 46 us; skew 500 ppm
Aug 06 17:02:29 dist-test-slave-69rg krb5kdc[12129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035749, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:29.602584 12142 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:02:29.603533 12142 webserver.cc:460] Webserver started at http://127.9.111.126:46409/ using document root <none> and password file <none>
I20260806 17:02:29.604168 12142 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:29.604274 12142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:29.604522 12142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:29.606769 12142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "fb20a33c752142a9aec89f1d39445a5d"
format_stamp: "Formatted at 2026-08-06 17:02:29 on dist-test-slave-69rg"
server_key: "47758595776a7024d16d3235c9ac56f7"
server_key_iv: "6ede7a8fbaef8a7e69af37ea25de4405"
server_key_version: "encryptionkey@0"
I20260806 17:02:29.607420 12142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "fb20a33c752142a9aec89f1d39445a5d"
format_stamp: "Formatted at 2026-08-06 17:02:29 on dist-test-slave-69rg"
server_key: "47758595776a7024d16d3235c9ac56f7"
server_key_iv: "6ede7a8fbaef8a7e69af37ea25de4405"
server_key_version: "encryptionkey@0"
I20260806 17:02:29.611383 12142 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:02:29.613924 12157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:29.615229 12142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 17:02:29.615424 12142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "fb20a33c752142a9aec89f1d39445a5d"
format_stamp: "Formatted at 2026-08-06 17:02:29 on dist-test-slave-69rg"
server_key: "47758595776a7024d16d3235c9ac56f7"
server_key_iv: "6ede7a8fbaef8a7e69af37ea25de4405"
server_key_version: "encryptionkey@0"
I20260806 17:02:29.615557 12142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:29.628139 12142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:29.632016 12142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:29.632308 12142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:29.640960 12142 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:45633
I20260806 17:02:29.640991 12210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:45633 every 8 connection(s)
I20260806 17:02:29.642165 12142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:02:29.645109 12211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:29.646848 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 12142
I20260806 17:02:29.646950 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:02:29.647228 9661 external_mini_cluster.cc:1428] Setting key 6d5fafbf5d405a0efb47181fe3867cdd
I20260806 17:02:29.652364 12211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d: Bootstrap starting.
I20260806 17:02:29.655146 12211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:29.656075 12211 log.cc:824] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d: Log is configured to *not* fsync() on all Append() calls
Aug 06 17:02:29 dist-test-slave-69rg krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:29.658747 12211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d: No bootstrap required, opened a new log
I20260806 17:02:29.663699 12211 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb20a33c752142a9aec89f1d39445a5d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45633 } }
I20260806 17:02:29.664026 12211 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:29.664075 12211 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb20a33c752142a9aec89f1d39445a5d, State: Initialized, Role: FOLLOWER
I20260806 17:02:29.664543 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:29.648794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.9.111.126:45633)
0806 17:02:29.649366 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:29.649387 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:29.649434 (+ 47us) server_negotiation.cc:407] Connection header received
0806 17:02:29.650382 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:29.650409 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:29.650855 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:29.651234 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:29.652011 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:29.653442 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:29.654412 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:29.655114 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:29.655287 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:29.658752 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:29.658760 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:29.658779 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:29.658814 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:29.660981 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:29.661488 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:29.661497 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:29.661503 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:29.661657 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:29.662053 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:29.662056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:29.662058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:29.662641 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:29.662868 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:29.663127 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 17:02:29.663467 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":185,"threads_started":1}
I20260806 17:02:29.664561 12211 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb20a33c752142a9aec89f1d39445a5d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45633 } }
I20260806 17:02:29.664736 12211 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:02:29.664849 12211 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:02:29.664986 12211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:29.666100 12211 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb20a33c752142a9aec89f1d39445a5d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45633 } }
I20260806 17:02:29.666461 12211 leader_election.cc:302] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb20a33c752142a9aec89f1d39445a5d; no voters:
I20260806 17:02:29.666826 12211 leader_election.cc:288] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:02:29.666952 12217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:29.667222 12217 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d [term 1 LEADER]: Becoming Leader. State: Replica: fb20a33c752142a9aec89f1d39445a5d, State: Running, Role: LEADER
I20260806 17:02:29.667575 12217 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb20a33c752142a9aec89f1d39445a5d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45633 } }
I20260806 17:02:29.668081 12211 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:29.669313 12219 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb20a33c752142a9aec89f1d39445a5d. Latest consensus state: current_term: 1 leader_uuid: "fb20a33c752142a9aec89f1d39445a5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb20a33c752142a9aec89f1d39445a5d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45633 } } }
I20260806 17:02:29.669458 12219 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:29.669799 12218 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb20a33c752142a9aec89f1d39445a5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb20a33c752142a9aec89f1d39445a5d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45633 } } }
I20260806 17:02:29.669970 12218 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:29.669996 12226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:29.673898 12226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:02:29.681367 12226 catalog_manager.cc:1347] Generated new cluster ID: 13dc4368a2164e18a35e173277678581
I20260806 17:02:29.681509 12226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:29.692400 12226 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:02:29.693382 12226 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:29.698894 12226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb20a33c752142a9aec89f1d39445a5d: Generated new TSK 0
I20260806 17:02:29.699555 12226 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
I20260806 17:02:29.735414 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:45633
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:29.857316 12238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:29.857700 12238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:29.857788 12238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:29.862293 12238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:29.862421 12238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:29.862524 12238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:02:29.868197 12238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:45633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:29.869469 12238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:29.870649 12238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:29.880939 12245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:29.880970 12247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:29.881342 12244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:29.882117 12238 server_base.cc:1034] running on GCE node
I20260806 17:02:29.882575 12238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:29.883342 12238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:29.884522 12238 hybrid_clock.cc:648] HybridClock initialized: now 1786035749884502 us; error 34 us; skew 500 ppm
Aug 06 17:02:29 dist-test-slave-69rg krb5kdc[12129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035749, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:29.887653 12238 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:02:29.888880 12238 webserver.cc:460] Webserver started at http://127.9.111.65:38001/ using document root <none> and password file <none>
I20260806 17:02:29.889490 12238 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:29.889592 12238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:29.889902 12238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:29.892055 12238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "1a8c91ea0a104222bfacb87f9aa3c759"
format_stamp: "Formatted at 2026-08-06 17:02:29 on dist-test-slave-69rg"
server_key: "1e85c8bb3e9adf522bc07ce5939cb7dc"
server_key_iv: "8f349e2a862cd04ff7d5168f410eb708"
server_key_version: "encryptionkey@0"
I20260806 17:02:29.892647 12238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "1a8c91ea0a104222bfacb87f9aa3c759"
format_stamp: "Formatted at 2026-08-06 17:02:29 on dist-test-slave-69rg"
server_key: "1e85c8bb3e9adf522bc07ce5939cb7dc"
server_key_iv: "8f349e2a862cd04ff7d5168f410eb708"
server_key_version: "encryptionkey@0"
I20260806 17:02:29.896571 12238 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260806 17:02:29.899374 12254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:29.900904 12238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 17:02:29.901078 12238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "1a8c91ea0a104222bfacb87f9aa3c759"
format_stamp: "Formatted at 2026-08-06 17:02:29 on dist-test-slave-69rg"
server_key: "1e85c8bb3e9adf522bc07ce5939cb7dc"
server_key_iv: "8f349e2a862cd04ff7d5168f410eb708"
server_key_version: "encryptionkey@0"
I20260806 17:02:29.901228 12238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:29.919442 12238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:29.922948 12238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:29.923207 12238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:29.923883 12238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:29.924823 12238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:29.924908 12238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:29.924983 12238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:29.925034 12238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:29.935019 12238 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:42719
I20260806 17:02:29.935073 12370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:42719 every 8 connection(s)
I20260806 17:02:29.936004 12238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:02:29.944029 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 12238
I20260806 17:02:29.944133 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:02:29.944417 9661 external_mini_cluster.cc:1428] Setting key 34afe29114b0f57801ea56cfb9b69df6
Aug 06 17:02:29 dist-test-slave-69rg krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035749, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:29.951809 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:29.937954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:33393 (local address 127.9.111.126:45633)
0806 17:02:29.938110 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:29.938114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:29.939024 (+ 910us) server_negotiation.cc:407] Connection header received
0806 17:02:29.940334 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:29.940338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:29.940400 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:29.940510 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:29.941728 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:29.942343 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:29.943495 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:29.943777 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:29.943906 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:29.947148 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:29.947156 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:29.947162 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:29.947198 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:29.948946 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:29.949697 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:29.949700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:29.949701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:29.949908 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:29.950395 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:29.950402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:29.950406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:29.950687 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:29.950827 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:29.951353 (+ 526us) server_negotiation.cc:299] Negotiation successful
0806 17:02:29.951530 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 17:02:29.952593 12373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:29.938396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:45633 (local address 127.9.111.65:33393)
0806 17:02:29.938872 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:29.938916 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:29.940112 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:29.940686 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:29.940692 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:29.941051 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:29.941571 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:29.941585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:29.942478 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:29.942487 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:29.943355 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:29.943365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:29.943960 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:29.943963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:29.944224 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:29.944935 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:29.944959 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:29.946965 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:29.949086 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:29.949094 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:29.949112 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:29.949559 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:29.950028 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:29.950035 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:29.950041 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:29.950286 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:29.950828 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:29.950833 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:29.951095 (+ 262us) client_negotiation.cc:769] Sending connection context
0806 17:02:29.951298 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 17:02:29.951584 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":124,"threads_started":1}
I20260806 17:02:29.953954 12371 heartbeater.cc:347] Connected to a master server at 127.9.111.126:45633
I20260806 17:02:29.954218 12371 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:29.954730 12371 heartbeater.cc:511] Master 127.9.111.126:45633 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.111.66@KRBTEST.COM; defaulting to no policy
I20260806 17:02:29.956467 12171 ts_manager.cc:194] Registered new tserver with Master: 1a8c91ea0a104222bfacb87f9aa3c759 (127.9.111.65:42719)
I20260806 17:02:29.957655 12171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:33393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
I20260806 17:02:29.977910 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:45633
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:30.101006 12376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:30.101320 12376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:30.101366 12376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:30.105511 12376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:30.105583 12376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:30.105646 12376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:02:30.111234 12376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:45633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:30.112432 12376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:30.113510 12376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:30.123422 12384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:30.123600 12382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:30.123422 12381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:30.123780 12376 server_base.cc:1034] running on GCE node
I20260806 17:02:30.124269 12376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:30.124902 12376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:30.126060 12376 hybrid_clock.cc:648] HybridClock initialized: now 1786035750126037 us; error 51 us; skew 500 ppm
Aug 06 17:02:30 dist-test-slave-69rg krb5kdc[12129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035750, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:30.128930 12376 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:02:30.131040 12376 webserver.cc:460] Webserver started at http://127.9.111.66:44523/ using document root <none> and password file <none>
I20260806 17:02:30.131659 12376 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:30.131743 12376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:30.131932 12376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:30.134299 12376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "b54334d5781a41959b70a96dae2de4b6"
format_stamp: "Formatted at 2026-08-06 17:02:30 on dist-test-slave-69rg"
server_key: "49605afda1ab35620b1bfb49d224fed9"
server_key_iv: "76f38fd97b524ac6823836c021a8f5ab"
server_key_version: "encryptionkey@0"
I20260806 17:02:30.134979 12376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "b54334d5781a41959b70a96dae2de4b6"
format_stamp: "Formatted at 2026-08-06 17:02:30 on dist-test-slave-69rg"
server_key: "49605afda1ab35620b1bfb49d224fed9"
server_key_iv: "76f38fd97b524ac6823836c021a8f5ab"
server_key_version: "encryptionkey@0"
I20260806 17:02:30.139240 12376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 17:02:30.142254 12391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:30.143639 12376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 17:02:30.143796 12376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "b54334d5781a41959b70a96dae2de4b6"
format_stamp: "Formatted at 2026-08-06 17:02:30 on dist-test-slave-69rg"
server_key: "49605afda1ab35620b1bfb49d224fed9"
server_key_iv: "76f38fd97b524ac6823836c021a8f5ab"
server_key_version: "encryptionkey@0"
I20260806 17:02:30.143930 12376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:30.162382 12376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:30.165385 12376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:30.165598 12376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:30.166277 12376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:30.167125 12376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:30.167196 12376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:30.167264 12376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:30.167311 12376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:30.177111 12376 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:43231
I20260806 17:02:30.177136 12511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:43231 every 8 connection(s)
I20260806 17:02:30.178133 12376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:02:30.186560 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 12376
I20260806 17:02:30.186705 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:02:30.187052 9661 external_mini_cluster.cc:1428] Setting key 634a70d78b811f482131d163f80ed4f3
Aug 06 17:02:30 dist-test-slave-69rg krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035750, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:30.193070 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:30.180251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:44943 (local address 127.9.111.126:45633)
0806 17:02:30.180398 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:30.180402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:30.181237 (+ 835us) server_negotiation.cc:407] Connection header received
0806 17:02:30.182563 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:30.182565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:30.182615 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:30.182693 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:30.183867 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:30.184514 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:30.185648 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:30.185993 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:30.186088 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:30.189134 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:30.189137 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:30.189140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:30.189165 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:30.190820 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:30.191370 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:30.191374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:30.191376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:30.191467 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:30.191842 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:30.191846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:30.191848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:30.192044 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:30.192154 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:30.192835 (+ 681us) server_negotiation.cc:299] Negotiation successful
0806 17:02:30.192949 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 17:02:30.194373 12514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:30.180625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:45633 (local address 127.9.111.66:44943)
0806 17:02:30.181067 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:30.181109 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:30.182354 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:30.182867 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:30.182873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:30.183229 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:30.183698 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:30.183711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:30.184633 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:30.184636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:30.185518 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:30.185527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:30.186130 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:30.186133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:30.186275 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:30.186998 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:30.187019 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:30.188920 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:30.190958 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:30.190964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:30.190980 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:30.191255 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:30.191572 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:30.191575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:30.191577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:30.191727 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:30.192154 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:30.192163 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:30.192503 (+ 340us) client_negotiation.cc:769] Sending connection context
0806 17:02:30.192749 (+ 246us) client_negotiation.cc:240] Negotiation successful
0806 17:02:30.193137 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":114,"threads_started":1}
I20260806 17:02:30.195757 12512 heartbeater.cc:347] Connected to a master server at 127.9.111.126:45633
I20260806 17:02:30.196025 12512 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:30.196547 12512 heartbeater.cc:511] Master 127.9.111.126:45633 requested a full tablet report, sending...
I20260806 17:02:30.197662 12171 ts_manager.cc:194] Registered new tserver with Master: b54334d5781a41959b70a96dae2de4b6 (127.9.111.66:43231)
I20260806 17:02:30.198323 12171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:44943
WARNING: no policy specified for kudu/127.9.111.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
I20260806 17:02:30.222860 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:45633
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:30.343019 12517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:30.343330 12517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:30.343415 12517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:30.347602 12517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:30.347697 12517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:30.347793 12517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:02:30.353276 12517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:45633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:30.354454 12517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:30.355542 12517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:30.365694 12522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:30.365727 12523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:30.365945 12517 server_base.cc:1034] running on GCE node
W20260806 17:02:30.365749 12525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:30.366565 12517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:30.367228 12517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:30.368458 12517 hybrid_clock.cc:648] HybridClock initialized: now 1786035750368437 us; error 39 us; skew 500 ppm
Aug 06 17:02:30 dist-test-slave-69rg krb5kdc[12129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035750, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:30.371594 12517 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:02:30.372495 12517 webserver.cc:460] Webserver started at http://127.9.111.67:43385/ using document root <none> and password file <none>
I20260806 17:02:30.373061 12517 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:30.373159 12517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:30.373389 12517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:30.375554 12517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "42f9d1b5005a4853a4d41844b0b89a0a"
format_stamp: "Formatted at 2026-08-06 17:02:30 on dist-test-slave-69rg"
server_key: "9c46e15264408694dc9a24b111755b05"
server_key_iv: "852727f77dcaea5583bb28133f1ed466"
server_key_version: "encryptionkey@0"
I20260806 17:02:30.376123 12517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "42f9d1b5005a4853a4d41844b0b89a0a"
format_stamp: "Formatted at 2026-08-06 17:02:30 on dist-test-slave-69rg"
server_key: "9c46e15264408694dc9a24b111755b05"
server_key_iv: "852727f77dcaea5583bb28133f1ed466"
server_key_version: "encryptionkey@0"
I20260806 17:02:30.379942 12517 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 17:02:30.382616 12532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:30.383847 12517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 17:02:30.383991 12517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "42f9d1b5005a4853a4d41844b0b89a0a"
format_stamp: "Formatted at 2026-08-06 17:02:30 on dist-test-slave-69rg"
server_key: "9c46e15264408694dc9a24b111755b05"
server_key_iv: "852727f77dcaea5583bb28133f1ed466"
server_key_version: "encryptionkey@0"
I20260806 17:02:30.384125 12517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:30.422510 12517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:30.426021 12517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:30.426240 12517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:30.426846 12517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:30.427675 12517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:30.427757 12517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:30.427846 12517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:30.427894 12517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:30.436998 12517 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:40033
I20260806 17:02:30.437024 12646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:40033 every 8 connection(s)
I20260806 17:02:30.438122 12517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:02:30.442531 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 12517
I20260806 17:02:30.442667 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:02:30.442996 9661 external_mini_cluster.cc:1428] Setting key b66ccb784e6aacbef6b00e9b3b5f712f
Aug 06 17:02:30 dist-test-slave-69rg krb5kdc[12129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035750, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:30.453765 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:30.440333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:37131 (local address 127.9.111.126:45633)
0806 17:02:30.440506 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:30.440510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:30.441443 (+ 933us) server_negotiation.cc:407] Connection header received
0806 17:02:30.443557 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:30.443596 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:30.443678 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:30.443771 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:30.445440 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:30.446068 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:30.447223 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:30.447530 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:30.447638 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:30.450252 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:30.450255 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:30.450257 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:30.450283 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:30.451671 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:30.452181 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:30.452185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:30.452186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:30.452273 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:30.452589 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:30.452591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:30.452592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:30.452789 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:30.452869 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:30.453499 (+ 630us) server_negotiation.cc:299] Negotiation successful
0806 17:02:30.453633 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 17:02:30.454669 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:30.440745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:45633 (local address 127.9.111.67:37131)
0806 17:02:30.441248 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:30.441307 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:30.443218 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:30.444050 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:30.444061 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:30.444593 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:30.445303 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:30.445320 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:30.446176 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:30.446179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:30.447074 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:30.447084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:30.447673 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:30.447676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:30.447843 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:30.448440 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:30.448457 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:30.450093 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:30.451788 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:30.451793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:30.451806 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:30.452065 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:30.452367 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:30.452370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:30.452371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:30.452507 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:30.452918 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:30.452924 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:30.453188 (+ 264us) client_negotiation.cc:769] Sending connection context
0806 17:02:30.453419 (+ 231us) client_negotiation.cc:240] Negotiation successful
0806 17:02:30.453707 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":118,"threads_started":1}
I20260806 17:02:30.455965 12647 heartbeater.cc:347] Connected to a master server at 127.9.111.126:45633
I20260806 17:02:30.456260 12647 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:30.456800 12647 heartbeater.cc:511] Master 127.9.111.126:45633 requested a full tablet report, sending...
I20260806 17:02:30.457957 12171 ts_manager.cc:194] Registered new tserver with Master: 42f9d1b5005a4853a4d41844b0b89a0a (127.9.111.67:40033)
I20260806 17:02:30.458516 12171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:37131
I20260806 17:02:30.469718 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 17:02:30.488112 12215 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:02:30.485990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45562 (local address 127.9.111.126:45633)
0806 17:02:30.486202 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:30.486209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:30.486420 (+ 211us) server_negotiation.cc:407] Connection header received
0806 17:02:30.486665 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:30.486668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:30.486720 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:30.486831 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:30.487942 (+ 1111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45562: BlockingRecv error: recv got EOF from 127.0.0.1:45562 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 17:02:30.490067 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 12238
I20260806 17:02:30.496599 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 12376
I20260806 17:02:30.504022 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 12517
I20260806 17:02:30.511487 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 12142
2026-08-06T17:02:30Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2276 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:02:30 dist-test-slave-69rg krb5kdc[12659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:02:30 dist-test-slave-69rg krb5kdc[12659](info): set up 2 sockets
Aug 06 17:02:30 dist-test-slave-69rg krb5kdc[12659](info): commencing operation
krb5kdc: starting...
W20260806 17:02:31.610379 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:02:31 dist-test-slave-69rg krb5kdc[12659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:02:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:02:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:02:31.749073 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:42741
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:46291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:42741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:31.862468 12672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:31.862798 12672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:31.862850 12672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:31.867106 12672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:31.867175 12672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:31.867197 12672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:31.867215 12672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:31.867283 12672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:02:31.872838 12672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:46291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:42741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:42741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:31.874007 12672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:31.875088 12672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:31.883746 12677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:31.883754 12678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:31.883754 12680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:31.884203 12672 server_base.cc:1034] running on GCE node
I20260806 17:02:31.884831 12672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:31.886067 12672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:31.887321 12672 hybrid_clock.cc:648] HybridClock initialized: now 1786035751887312 us; error 33 us; skew 500 ppm
Aug 06 17:02:31 dist-test-slave-69rg krb5kdc[12659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035751, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:31.890194 12672 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:02:31.891168 12672 webserver.cc:460] Webserver started at http://127.9.111.126:41791/ using document root <none> and password file <none>
I20260806 17:02:31.891697 12672 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:31.891758 12672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:31.891927 12672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:31.894060 12672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "b6ae59d4894648a297e42a46aeeda595"
format_stamp: "Formatted at 2026-08-06 17:02:31 on dist-test-slave-69rg"
server_key: "8c5339447e20944efdccb4932cea6926"
server_key_iv: "b46ec19b2a5d98c9ae8b8287662c5b73"
server_key_version: "encryptionkey@0"
I20260806 17:02:31.894613 12672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "b6ae59d4894648a297e42a46aeeda595"
format_stamp: "Formatted at 2026-08-06 17:02:31 on dist-test-slave-69rg"
server_key: "8c5339447e20944efdccb4932cea6926"
server_key_iv: "b46ec19b2a5d98c9ae8b8287662c5b73"
server_key_version: "encryptionkey@0"
I20260806 17:02:31.898339 12672 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 17:02:31.900629 12688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:31.901978 12672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:02:31.902119 12672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "b6ae59d4894648a297e42a46aeeda595"
format_stamp: "Formatted at 2026-08-06 17:02:31 on dist-test-slave-69rg"
server_key: "8c5339447e20944efdccb4932cea6926"
server_key_iv: "b46ec19b2a5d98c9ae8b8287662c5b73"
server_key_version: "encryptionkey@0"
I20260806 17:02:31.902240 12672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:31.925940 12672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:31.929257 12672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:31.929490 12672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:31.936714 12672 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:42741
I20260806 17:02:31.937038 12740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:42741 every 8 connection(s)
I20260806 17:02:31.937706 12672 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:02:31.940555 12741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:31.946147 12741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595: Bootstrap starting.
I20260806 17:02:31.946729 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 12672
I20260806 17:02:31.946862 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:02:31.947153 9661 external_mini_cluster.cc:1428] Setting key a679136e540abe64d7e69eb906c0430c
I20260806 17:02:31.948949 12741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:31.949720 12741 log.cc:824] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:31.951642 12741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595: No bootstrap required, opened a new log
I20260806 17:02:31.955386 12741 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ae59d4894648a297e42a46aeeda595" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42741 } }
I20260806 17:02:31.955619 12741 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:31.955704 12741 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ae59d4894648a297e42a46aeeda595, State: Initialized, Role: FOLLOWER
Aug 06 17:02:31 dist-test-slave-69rg krb5kdc[12659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:31.956126 12741 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ae59d4894648a297e42a46aeeda595" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42741 } }
I20260806 17:02:31.956311 12741 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:02:31.956382 12741 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:02:31.956477 12741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:31.957422 12741 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ae59d4894648a297e42a46aeeda595" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42741 } }
I20260806 17:02:31.957772 12741 leader_election.cc:302] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6ae59d4894648a297e42a46aeeda595; no voters:
I20260806 17:02:31.958250 12741 leader_election.cc:288] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:02:31.958393 12746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:31.958621 12746 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595 [term 1 LEADER]: Becoming Leader. State: Replica: b6ae59d4894648a297e42a46aeeda595, State: Running, Role: LEADER
I20260806 17:02:31.958933 12746 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ae59d4894648a297e42a46aeeda595" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42741 } }
I20260806 17:02:31.959648 12741 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:31.961004 12747 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6ae59d4894648a297e42a46aeeda595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ae59d4894648a297e42a46aeeda595" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42741 } } }
I20260806 17:02:31.961138 12747 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:31.962255 12748 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6ae59d4894648a297e42a46aeeda595. Latest consensus state: current_term: 1 leader_uuid: "b6ae59d4894648a297e42a46aeeda595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ae59d4894648a297e42a46aeeda595" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42741 } } }
I20260806 17:02:31.962360 12748 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:31.962469 12755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:31.963470 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:31.948457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34964 (local address 127.9.111.126:42741)
0806 17:02:31.948869 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:31.948896 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:31.948933 (+ 37us) server_negotiation.cc:407] Connection header received
0806 17:02:31.949678 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:31.949697 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:31.950123 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:31.950469 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:31.951346 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:31.952358 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:31.953236 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:31.953726 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:31.953943 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:31.956835 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:31.956842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:31.956865 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:31.956894 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:31.959161 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:31.959703 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:31.959712 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:31.959719 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:31.959870 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:31.960231 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:31.960235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:31.960237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:31.960835 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:31.961900 (+ 1065us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:31.962054 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 17:02:31.962478 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":141,"threads_started":1}
I20260806 17:02:31.965361 12755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:02:31.966871 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:02:31.971529 12755 catalog_manager.cc:1347] Generated new cluster ID: 22971fbb574b434384fe7f2149561eff
I20260806 17:02:31.971632 12755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:31.987991 12755 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:02:31.989261 12755 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:31.989403 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:31.984008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.9.111.126:42741)
0806 17:02:31.984150 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:31.984156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:31.984414 (+ 258us) server_negotiation.cc:407] Connection header received
0806 17:02:31.984591 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:31.984594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:31.984651 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:31.984742 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:31.985319 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:31.985895 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:31.986892 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:31.987263 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:31.987377 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:31.988391 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:31.988395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:31.988398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:02:31.988871 (+ 473us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:31.988903 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:02:31.988920 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:31.989031 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:31.989125 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 17:02:31.989191 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 17:02:31.993858 12755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6ae59d4894648a297e42a46aeeda595: Generated new TSK 0
I20260806 17:02:31.994629 12755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:02:32.004761 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 12672
2026-08-06T17:02:32Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1491 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:02:32 dist-test-slave-69rg krb5kdc[12774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:02:32 dist-test-slave-69rg krb5kdc[12774](info): set up 2 sockets
Aug 06 17:02:32 dist-test-slave-69rg krb5kdc[12774](info): commencing operation
krb5kdc: starting...
W20260806 17:02:33.077797 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:02:33 dist-test-slave-69rg krb5kdc[12774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:02:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:02:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:02:33.221318 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:37181
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:40821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:37181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:33.336489 12787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:33.336861 12787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:33.336949 12787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:33.341137 12787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:33.341241 12787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:33.341277 12787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:33.341315 12787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:33.341374 12787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:02:33.346834 12787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:40821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:37181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:37181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:33.348071 12787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:33.349165 12787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:33.358160 12792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:33.358240 12787 server_base.cc:1034] running on GCE node
W20260806 17:02:33.358160 12793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:33.358311 12795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:33.359081 12787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:33.360174 12787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:33.361375 12787 hybrid_clock.cc:648] HybridClock initialized: now 1786035753361360 us; error 34 us; skew 500 ppm
Aug 06 17:02:33 dist-test-slave-69rg krb5kdc[12774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035753, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:33.368822 12787 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:02:33.369758 12787 webserver.cc:460] Webserver started at http://127.9.111.126:39019/ using document root <none> and password file <none>
I20260806 17:02:33.370383 12787 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:33.370489 12787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:33.370734 12787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:33.372812 12787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "1038f49b289f4f16bc6b31b879c5ca23"
format_stamp: "Formatted at 2026-08-06 17:02:33 on dist-test-slave-69rg"
server_key: "a67c65efa491054b8be08baa7f9d16e3"
server_key_iv: "8d34187416fe2898fc3cd9fd65be7c4d"
server_key_version: "encryptionkey@0"
I20260806 17:02:33.373407 12787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "1038f49b289f4f16bc6b31b879c5ca23"
format_stamp: "Formatted at 2026-08-06 17:02:33 on dist-test-slave-69rg"
server_key: "a67c65efa491054b8be08baa7f9d16e3"
server_key_iv: "8d34187416fe2898fc3cd9fd65be7c4d"
server_key_version: "encryptionkey@0"
I20260806 17:02:33.377110 12787 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 17:02:33.379681 12803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:33.380995 12787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 17:02:33.381139 12787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "1038f49b289f4f16bc6b31b879c5ca23"
format_stamp: "Formatted at 2026-08-06 17:02:33 on dist-test-slave-69rg"
server_key: "a67c65efa491054b8be08baa7f9d16e3"
server_key_iv: "8d34187416fe2898fc3cd9fd65be7c4d"
server_key_version: "encryptionkey@0"
I20260806 17:02:33.381266 12787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:33.414915 12787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:33.418614 12787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:33.418849 12787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:33.426477 12787 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:37181
I20260806 17:02:33.426805 12856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:37181 every 8 connection(s)
I20260806 17:02:33.427600 12787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:02:33.429960 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 12787
I20260806 17:02:33.430058 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:02:33.430357 9661 external_mini_cluster.cc:1428] Setting key 8c564fc58ebb2f61a1caa18055b73cc9
I20260806 17:02:33.430662 12857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:33.437997 12857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23: Bootstrap starting.
Aug 06 17:02:33 dist-test-slave-69rg krb5kdc[12774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:33.440753 12857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:33.441747 12857 log.cc:824] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:33.444276 12857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23: No bootstrap required, opened a new log
I20260806 17:02:33.446409 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:33.431804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48132 (local address 127.9.111.126:37181)
0806 17:02:33.432225 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:33.432245 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:33.432286 (+ 41us) server_negotiation.cc:407] Connection header received
0806 17:02:33.433325 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:33.433350 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:33.433779 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:33.434177 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:33.434883 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:33.436369 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:33.437211 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:33.437924 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:33.438108 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:33.441092 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:33.441098 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:33.441116 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:33.441150 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:33.443130 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:33.443649 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:33.443656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:33.443662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:33.443806 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:33.444091 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:33.444094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:33.444095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:33.444629 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:33.444813 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:33.445092 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 17:02:33.445399 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":106,"threads_started":1}
I20260806 17:02:33.448802 12857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1038f49b289f4f16bc6b31b879c5ca23" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37181 } }
I20260806 17:02:33.449033 12857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:33.449067 12857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1038f49b289f4f16bc6b31b879c5ca23, State: Initialized, Role: FOLLOWER
I20260806 17:02:33.449419 12857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1038f49b289f4f16bc6b31b879c5ca23" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37181 } }
I20260806 17:02:33.449527 12857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:02:33.449563 12857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:02:33.449631 12857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:33.450593 12857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1038f49b289f4f16bc6b31b879c5ca23" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37181 } }
I20260806 17:02:33.450887 12857 leader_election.cc:302] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1038f49b289f4f16bc6b31b879c5ca23; no voters:
I20260806 17:02:33.451251 12857 leader_election.cc:288] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:02:33.451527 12862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:33.451722 12862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23 [term 1 LEADER]: Becoming Leader. State: Replica: 1038f49b289f4f16bc6b31b879c5ca23, State: Running, Role: LEADER
I20260806 17:02:33.452011 12862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1038f49b289f4f16bc6b31b879c5ca23" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37181 } }
I20260806 17:02:33.452574 12857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:33.453701 12864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1038f49b289f4f16bc6b31b879c5ca23. Latest consensus state: current_term: 1 leader_uuid: "1038f49b289f4f16bc6b31b879c5ca23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1038f49b289f4f16bc6b31b879c5ca23" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37181 } } }
I20260806 17:02:33.453860 12864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:33.454103 12863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1038f49b289f4f16bc6b31b879c5ca23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1038f49b289f4f16bc6b31b879c5ca23" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37181 } } }
I20260806 17:02:33.454210 12863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:33.456099 12878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:33.459117 12878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:02:33.464664 12878 catalog_manager.cc:1347] Generated new cluster ID: 990afca43b524d9ea3d53fecb5f61d90
I20260806 17:02:33.464740 12878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:33.473126 12878 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:02:33.474133 12878 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:33.490268 12878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1038f49b289f4f16bc6b31b879c5ca23: Generated new TSK 0
I20260806 17:02:33.491041 12878 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
I20260806 17:02:33.529973 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:37181
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:40821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:33.659855 12884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:33.660192 12884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:33.660240 12884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:33.665076 12884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:33.665184 12884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:33.665251 12884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:02:33.671393 12884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:40821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:37181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:33.672657 12884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:33.673940 12884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:33.685509 12889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:33.685490 12890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:33.685490 12892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:33.686039 12884 server_base.cc:1034] running on GCE node
I20260806 17:02:33.686785 12884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:33.687686 12884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:33.688930 12884 hybrid_clock.cc:648] HybridClock initialized: now 1786035753688890 us; error 56 us; skew 500 ppm
Aug 06 17:02:33 dist-test-slave-69rg krb5kdc[12774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035753, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:33.692076 12884 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:02:33.693159 12884 webserver.cc:460] Webserver started at http://127.9.111.65:33561/ using document root <none> and password file <none>
I20260806 17:02:33.693782 12884 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:33.693933 12884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:33.694182 12884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:33.696382 12884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "b4ed628f9248435c9c7929fc8e4bdd46"
format_stamp: "Formatted at 2026-08-06 17:02:33 on dist-test-slave-69rg"
server_key: "ed95276f89b1e57e83e59d76e9b77a50"
server_key_iv: "8f3c3238ed44e8f5d45f5b0aa393537b"
server_key_version: "encryptionkey@0"
I20260806 17:02:33.697301 12884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "b4ed628f9248435c9c7929fc8e4bdd46"
format_stamp: "Formatted at 2026-08-06 17:02:33 on dist-test-slave-69rg"
server_key: "ed95276f89b1e57e83e59d76e9b77a50"
server_key_iv: "8f3c3238ed44e8f5d45f5b0aa393537b"
server_key_version: "encryptionkey@0"
I20260806 17:02:33.701942 12884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260806 17:02:33.704823 12899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:33.706405 12884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:33.706565 12884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "b4ed628f9248435c9c7929fc8e4bdd46"
format_stamp: "Formatted at 2026-08-06 17:02:33 on dist-test-slave-69rg"
server_key: "ed95276f89b1e57e83e59d76e9b77a50"
server_key_iv: "8f3c3238ed44e8f5d45f5b0aa393537b"
server_key_version: "encryptionkey@0"
I20260806 17:02:33.706697 12884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:33.730440 12884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:33.733855 12884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:33.734084 12884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:33.734723 12884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:33.735635 12884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:33.735709 12884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:33.735780 12884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:33.735823 12884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:33.746464 12884 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:38497
I20260806 17:02:33.746490 13015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:38497 every 8 connection(s)
I20260806 17:02:33.747560 12884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:02:33.748785 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 12884
I20260806 17:02:33.748935 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:02:33.749264 9661 external_mini_cluster.cc:1428] Setting key c7bf0d45a39bcf54a9cfb75cc39d507a
Aug 06 17:02:33 dist-test-slave-69rg krb5kdc[12774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035753, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.111.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.66@KRBTEST.COM" created.
I20260806 17:02:33.767562 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:33.750341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:44227 (local address 127.9.111.126:37181)
0806 17:02:33.750940 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:33.750946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:33.751558 (+ 612us) server_negotiation.cc:407] Connection header received
0806 17:02:33.753955 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:33.753959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:33.754027 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:33.754112 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:33.755861 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:33.756418 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:33.758535 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:33.758792 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:33.758881 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:33.762490 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:33.762494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:33.762496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:33.762555 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:33.764509 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:33.765241 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:33.765245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:33.765246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:33.765365 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:33.765803 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:33.765806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:33.765808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:33.766182 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:33.766348 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:33.767070 (+ 722us) server_negotiation.cc:299] Negotiation successful
0806 17:02:33.767291 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509}
I20260806 17:02:33.768491 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:33.750693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37181 (local address 127.9.111.65:44227)
0806 17:02:33.751356 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:33.751427 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:33.753444 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:33.754395 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:33.754409 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:33.755046 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:33.755678 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:33.755693 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:33.756576 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:33.756583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:33.758351 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:33.758368 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:33.758997 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:33.759004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:33.759295 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:33.760141 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:33.760167 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:33.762240 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:33.764678 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:33.764697 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:33.764713 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:33.765036 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:33.765475 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:33.765478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:33.765480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:33.765659 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:33.766385 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:33.766397 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:33.766771 (+ 374us) client_negotiation.cc:769] Sending connection context
0806 17:02:33.766969 (+ 198us) client_negotiation.cc:240] Negotiation successful
0806 17:02:33.767282 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":203,"threads_started":1}
I20260806 17:02:33.770118 13016 heartbeater.cc:347] Connected to a master server at 127.9.111.126:37181
I20260806 17:02:33.770413 13016 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:33.770967 13016 heartbeater.cc:511] Master 127.9.111.126:37181 requested a full tablet report, sending...
I20260806 17:02:33.772918 12820 ts_manager.cc:194] Registered new tserver with Master: b4ed628f9248435c9c7929fc8e4bdd46 (127.9.111.65:38497)
I20260806 17:02:33.774832 12820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:44227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
I20260806 17:02:33.795145 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:37181
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:40821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:33.907753 13021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:33.908073 13021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:33.908155 13021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:33.912379 13021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:33.912470 13021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:33.912560 13021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:02:33.917742 13021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:40821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:37181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:33.918880 13021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:33.919914 13021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:33.929988 13027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:33.930006 13030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:33.930006 13028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:33.930086 13021 server_base.cc:1034] running on GCE node
I20260806 17:02:33.930658 13021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:33.931264 13021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:33.932461 13021 hybrid_clock.cc:648] HybridClock initialized: now 1786035753932452 us; error 33 us; skew 500 ppm
Aug 06 17:02:33 dist-test-slave-69rg krb5kdc[12774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035753, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:33.935442 13021 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:02:33.936358 13021 webserver.cc:460] Webserver started at http://127.9.111.66:41819/ using document root <none> and password file <none>
I20260806 17:02:33.936931 13021 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:33.937031 13021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:33.937282 13021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:33.939424 13021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "8074ca53f41048fe8521890ab0a9da5c"
format_stamp: "Formatted at 2026-08-06 17:02:33 on dist-test-slave-69rg"
server_key: "f34f4b69821add3229fd7c7e10dc7941"
server_key_iv: "bbe781c6382029527fd73ac1fa5b92da"
server_key_version: "encryptionkey@0"
I20260806 17:02:33.940017 13021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "8074ca53f41048fe8521890ab0a9da5c"
format_stamp: "Formatted at 2026-08-06 17:02:33 on dist-test-slave-69rg"
server_key: "f34f4b69821add3229fd7c7e10dc7941"
server_key_iv: "bbe781c6382029527fd73ac1fa5b92da"
server_key_version: "encryptionkey@0"
I20260806 17:02:33.943900 13021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 17:02:33.946539 13037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:33.947993 13021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 17:02:33.948156 13021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "8074ca53f41048fe8521890ab0a9da5c"
format_stamp: "Formatted at 2026-08-06 17:02:33 on dist-test-slave-69rg"
server_key: "f34f4b69821add3229fd7c7e10dc7941"
server_key_iv: "bbe781c6382029527fd73ac1fa5b92da"
server_key_version: "encryptionkey@0"
I20260806 17:02:33.948283 13021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:33.984401 13021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:33.987972 13021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:33.988206 13021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:33.988832 13021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:33.989748 13021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:33.989820 13021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:33.989910 13021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:33.989959 13021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:33.999225 13021 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:43701
I20260806 17:02:33.999256 13154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:43701 every 8 connection(s)
I20260806 17:02:34.000118 13021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:02:34.003580 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 13021
I20260806 17:02:34.003716 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:02:34.004060 9661 external_mini_cluster.cc:1428] Setting key d9656143a830f71803d756543af6536b
Aug 06 17:02:34 dist-test-slave-69rg krb5kdc[12774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035753, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.111.67@KRBTEST.COM; defaulting to no policy
I20260806 17:02:34.016567 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:34.002203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:55979 (local address 127.9.111.126:37181)
0806 17:02:34.002372 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:34.002377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:34.003271 (+ 894us) server_negotiation.cc:407] Connection header received
0806 17:02:34.005039 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:34.005042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:34.005107 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:34.005214 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:34.006715 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:34.007236 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:34.008727 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:34.008957 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:34.009077 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:34.012387 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:34.012390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:34.012393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:34.012421 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:34.014126 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:34.014835 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:34.014838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:34.014839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:34.014929 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:34.015375 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:34.015377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:34.015378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:34.015649 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:34.015735 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:34.016284 (+ 549us) server_negotiation.cc:299] Negotiation successful
0806 17:02:34.016409 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 17:02:34.017482 13157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:34.002598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37181 (local address 127.9.111.66:55979)
0806 17:02:34.003105 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:34.003150 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:34.004821 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:34.005389 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:34.005396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:34.005811 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:34.006522 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:34.006545 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:34.007408 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:34.007415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:34.008595 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:34.008605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:34.009121 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:34.009125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:34.009306 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:34.010126 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:34.010143 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:34.012129 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:34.014257 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:34.014265 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:34.014282 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:34.014678 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:34.015039 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:34.015044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:34.015048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:34.015268 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:34.015775 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:34.015781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:34.016050 (+ 269us) client_negotiation.cc:769] Sending connection context
0806 17:02:34.016223 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 17:02:34.016586 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.67@KRBTEST.COM" created.
I20260806 17:02:34.019013 13155 heartbeater.cc:347] Connected to a master server at 127.9.111.126:37181
I20260806 17:02:34.019306 13155 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:34.019856 13155 heartbeater.cc:511] Master 127.9.111.126:37181 requested a full tablet report, sending...
I20260806 17:02:34.021006 12820 ts_manager.cc:194] Registered new tserver with Master: 8074ca53f41048fe8521890ab0a9da5c (127.9.111.66:43701)
I20260806 17:02:34.021615 12820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:55979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
I20260806 17:02:34.037535 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:37181
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:40821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:34.164608 13160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:34.164904 13160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:34.164970 13160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:34.169606 13160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:34.169704 13160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:34.169775 13160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:02:34.176481 13160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:40821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:37181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:34.177646 13160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:34.178812 13160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:34.189363 13168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:34.189350 13166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:34.189368 13165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:34.189872 13160 server_base.cc:1034] running on GCE node
I20260806 17:02:34.190248 13160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:34.190860 13160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:34.192026 13160 hybrid_clock.cc:648] HybridClock initialized: now 1786035754192008 us; error 30 us; skew 500 ppm
Aug 06 17:02:34 dist-test-slave-69rg krb5kdc[12774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035754, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:34.194970 13160 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:02:34.195896 13160 webserver.cc:460] Webserver started at http://127.9.111.67:39841/ using document root <none> and password file <none>
I20260806 17:02:34.196462 13160 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:34.196555 13160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:34.196859 13160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:34.199035 13160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "6ae0328d1ec54f478225948299be96fc"
format_stamp: "Formatted at 2026-08-06 17:02:34 on dist-test-slave-69rg"
server_key: "7e38a32970cf4c96c9cc0e34ea151eba"
server_key_iv: "ac4c69a1e9947d10d4b6f765c913b1e2"
server_key_version: "encryptionkey@0"
I20260806 17:02:34.199642 13160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "6ae0328d1ec54f478225948299be96fc"
format_stamp: "Formatted at 2026-08-06 17:02:34 on dist-test-slave-69rg"
server_key: "7e38a32970cf4c96c9cc0e34ea151eba"
server_key_iv: "ac4c69a1e9947d10d4b6f765c913b1e2"
server_key_version: "encryptionkey@0"
I20260806 17:02:34.203509 13160 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 17:02:34.206497 13176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:34.208212 13160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:34.208411 13160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "6ae0328d1ec54f478225948299be96fc"
format_stamp: "Formatted at 2026-08-06 17:02:34 on dist-test-slave-69rg"
server_key: "7e38a32970cf4c96c9cc0e34ea151eba"
server_key_iv: "ac4c69a1e9947d10d4b6f765c913b1e2"
server_key_version: "encryptionkey@0"
I20260806 17:02:34.208564 13160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:34.226037 13160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:34.229702 13160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:34.229948 13160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:34.230572 13160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:34.231424 13160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:34.231505 13160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:34.231580 13160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:34.231634 13160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:34.240697 13160 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:39069
I20260806 17:02:34.240732 13291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:39069 every 8 connection(s)
I20260806 17:02:34.241611 13160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:02:34.246022 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 13160
I20260806 17:02:34.246181 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:02:34.246476 9661 external_mini_cluster.cc:1428] Setting key 541289035ae566bce3e6241ec03f3490
Aug 06 17:02:34 dist-test-slave-69rg krb5kdc[12774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035754, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:34.257141 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:34.243561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:34305 (local address 127.9.111.126:37181)
0806 17:02:34.243744 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:34.243749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:34.244548 (+ 799us) server_negotiation.cc:407] Connection header received
0806 17:02:34.245958 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:34.245964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:34.246028 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:34.246135 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:34.247632 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:34.248259 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:34.249633 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:34.249928 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:34.250021 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:34.253080 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:34.253083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:34.253085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:34.253123 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:34.254853 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:34.255479 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:34.255482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:34.255483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:34.255571 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:34.255956 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:34.255960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:34.255962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:34.256177 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:34.256293 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:34.256830 (+ 537us) server_negotiation.cc:299] Negotiation successful
0806 17:02:34.256971 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 17:02:34.258108 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:34.243936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37181 (local address 127.9.111.67:34305)
0806 17:02:34.244393 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:34.244438 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:34.245681 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:34.246391 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:34.246397 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:34.246819 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:34.247493 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:34.247510 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:34.248411 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:34.248413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:34.249493 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:34.249508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:34.250107 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:34.250110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:34.250273 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:34.251008 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:34.251035 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:34.252870 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:34.255012 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:34.255029 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:34.255048 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:34.255342 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:34.255673 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:34.255677 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:34.255679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:34.255848 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:34.256293 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:34.256300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:34.256579 (+ 279us) client_negotiation.cc:769] Sending connection context
0806 17:02:34.256783 (+ 204us) client_negotiation.cc:240] Negotiation successful
0806 17:02:34.257145 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":119,"threads_started":1}
I20260806 17:02:34.259375 13292 heartbeater.cc:347] Connected to a master server at 127.9.111.126:37181
I20260806 17:02:34.259657 13292 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:34.260268 13292 heartbeater.cc:511] Master 127.9.111.126:37181 requested a full tablet report, sending...
I20260806 17:02:34.261408 12820 ts_manager.cc:194] Registered new tserver with Master: 6ae0328d1ec54f478225948299be96fc (127.9.111.67:39069)
I20260806 17:02:34.262077 12820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:34305
I20260806 17:02:34.272779 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 17:02:34 dist-test-slave-69rg krb5kdc[12774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035754, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 17:02:34 dist-test-slave-69rg krb5kdc[12774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035754, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:34.301090 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:34.291903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48134 (local address 127.9.111.126:37181)
0806 17:02:34.292101 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:34.292106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:34.292250 (+ 144us) server_negotiation.cc:407] Connection header received
0806 17:02:34.292431 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:34.292435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:34.292481 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:34.292573 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:34.293383 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:34.293991 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:34.294846 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:34.295071 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:34.295188 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:34.297732 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:34.297735 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:34.297737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:34.297765 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:34.299245 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:34.299699 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:34.299703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:34.299705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:34.299797 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:34.300085 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:34.300089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:34.300091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:34.300377 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:34.300532 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:34.300776 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 17:02:34.300910 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260806 17:02:34.301600 12820 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48134
I20260806 17:02:34.302574 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 12884
I20260806 17:02:34.308488 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 13021
I20260806 17:02:34.315235 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 13160
I20260806 17:02:34.321702 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 12787
2026-08-06T17:02:34Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2317 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:02:34 dist-test-slave-69rg krb5kdc[13304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:02:34 dist-test-slave-69rg krb5kdc[13304](info): set up 2 sockets
Aug 06 17:02:34 dist-test-slave-69rg krb5kdc[13304](info): commencing operation
krb5kdc: starting...
W20260806 17:02:35.417948 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:02:35 dist-test-slave-69rg krb5kdc[13304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:02:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:02:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:02:35.552726 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:33525
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:42971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:33525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:35.672991 13318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:35.673365 13318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:35.673454 13318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:35.677655 13318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:35.677778 13318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:35.677814 13318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:35.677868 13318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:35.677903 13318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:02:35.686790 13318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:42971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:33525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:33525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:35.688016 13318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:35.689074 13318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:35.699115 13326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:35.699115 13324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:35.699136 13323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:35.699393 13318 server_base.cc:1034] running on GCE node
I20260806 17:02:35.700109 13318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:35.701318 13318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:35.702661 13318 hybrid_clock.cc:648] HybridClock initialized: now 1786035755702624 us; error 43 us; skew 500 ppm
Aug 06 17:02:35 dist-test-slave-69rg krb5kdc[13304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035755, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:35.705917 13318 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:02:35.706971 13318 webserver.cc:460] Webserver started at http://127.9.111.126:40781/ using document root <none> and password file <none>
I20260806 17:02:35.707589 13318 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:35.707695 13318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:35.707953 13318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:35.710234 13318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "916bab1d7d924fe59894be635630f80a"
format_stamp: "Formatted at 2026-08-06 17:02:35 on dist-test-slave-69rg"
server_key: "442369cd8fe773644c919ee90e9fe96d"
server_key_iv: "57e78006ef2d1e6a24cde0c362cc041e"
server_key_version: "encryptionkey@0"
I20260806 17:02:35.710834 13318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "916bab1d7d924fe59894be635630f80a"
format_stamp: "Formatted at 2026-08-06 17:02:35 on dist-test-slave-69rg"
server_key: "442369cd8fe773644c919ee90e9fe96d"
server_key_iv: "57e78006ef2d1e6a24cde0c362cc041e"
server_key_version: "encryptionkey@0"
I20260806 17:02:35.714849 13318 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:02:35.717492 13333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:35.719630 13318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:35.719843 13318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "916bab1d7d924fe59894be635630f80a"
format_stamp: "Formatted at 2026-08-06 17:02:35 on dist-test-slave-69rg"
server_key: "442369cd8fe773644c919ee90e9fe96d"
server_key_iv: "57e78006ef2d1e6a24cde0c362cc041e"
server_key_version: "encryptionkey@0"
I20260806 17:02:35.719993 13318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:35.740581 13318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:35.744153 13318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:35.744434 13318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:35.753548 13318 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:33525
I20260806 17:02:35.753574 13388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:33525 every 8 connection(s)
I20260806 17:02:35.754696 13318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:02:35.757921 13389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:35.761133 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 13318
I20260806 17:02:35.761304 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:02:35.761644 9661 external_mini_cluster.cc:1428] Setting key 6e0943e7a5cd594e66bbb4c324b5c347
I20260806 17:02:35.766103 13389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a: Bootstrap starting.
I20260806 17:02:35.769176 13389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:35.770174 13389 log.cc:824] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a: Log is configured to *not* fsync() on all Append() calls
Aug 06 17:02:35 dist-test-slave-69rg krb5kdc[13304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:35.772677 13389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a: No bootstrap required, opened a new log
I20260806 17:02:35.778117 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:35.763502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37426 (local address 127.9.111.126:33525)
0806 17:02:35.764024 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:35.764041 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:35.764078 (+ 37us) server_negotiation.cc:407] Connection header received
0806 17:02:35.764914 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:35.764934 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:35.765323 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:35.765646 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:35.766779 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:35.767831 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:35.769071 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:35.769648 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:35.769798 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:35.772808 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:35.772815 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:35.772835 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:35.772873 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:35.774856 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:35.775356 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:35.775361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:35.775366 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:35.775483 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:35.775800 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:35.775804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:35.775806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:35.776230 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:35.776463 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:35.776756 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 17:02:35.777218 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":164,"threads_started":1}
I20260806 17:02:35.779100 13389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916bab1d7d924fe59894be635630f80a" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 33525 } }
I20260806 17:02:35.779438 13389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:35.779506 13389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916bab1d7d924fe59894be635630f80a, State: Initialized, Role: FOLLOWER
I20260806 17:02:35.780092 13389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916bab1d7d924fe59894be635630f80a" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 33525 } }
I20260806 17:02:35.780318 13389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:02:35.780395 13389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:02:35.780536 13389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:35.881075 13389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916bab1d7d924fe59894be635630f80a" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 33525 } }
I20260806 17:02:35.881697 13389 leader_election.cc:302] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 916bab1d7d924fe59894be635630f80a; no voters:
I20260806 17:02:35.882287 13389 leader_election.cc:288] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:02:35.882684 13394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:35.883189 13394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a [term 1 LEADER]: Becoming Leader. State: Replica: 916bab1d7d924fe59894be635630f80a, State: Running, Role: LEADER
I20260806 17:02:35.883998 13389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:35.883920 13394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916bab1d7d924fe59894be635630f80a" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 33525 } }
I20260806 17:02:35.887367 13397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 916bab1d7d924fe59894be635630f80a. Latest consensus state: current_term: 1 leader_uuid: "916bab1d7d924fe59894be635630f80a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916bab1d7d924fe59894be635630f80a" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 33525 } } }
I20260806 17:02:35.887506 13397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:35.887676 13395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "916bab1d7d924fe59894be635630f80a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916bab1d7d924fe59894be635630f80a" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 33525 } } }
I20260806 17:02:35.887809 13395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a [sys.catalog]: This master's current role is: LEADER
W20260806 17:02:35.890411 13409 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:02:35.890507 13409 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:02:35.890570 13410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:35.891462 13410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:02:35.898489 13410 catalog_manager.cc:1347] Generated new cluster ID: 3f49ba0f2fc047e1840887fed088112a
I20260806 17:02:35.898566 13410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:35.907404 13410 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:02:35.908680 13410 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:35.922863 13410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 916bab1d7d924fe59894be635630f80a: Generated new TSK 0
I20260806 17:02:35.923817 13410 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
W20260806 17:02:37.069096 9661 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.9.111.65: real 1.137s user 0.002s sys 0.003s
I20260806 17:02:37.069675 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:33525
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:42971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:37.186530 13417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:37.186856 13417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:37.186902 13417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:37.191069 13417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:37.191152 13417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:37.191216 13417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:02:37.196863 13417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:42971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:33525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:37.198021 13417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:37.199061 13417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:37.209161 13422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:37.209125 13423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:37.209152 13425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:37.209581 13417 server_base.cc:1034] running on GCE node
I20260806 17:02:37.210100 13417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:37.210842 13417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:37.212069 13417 hybrid_clock.cc:648] HybridClock initialized: now 1786035757212024 us; error 56 us; skew 500 ppm
Aug 06 17:02:37 dist-test-slave-69rg krb5kdc[13304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:37.215262 13417 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:02:37.216324 13417 webserver.cc:460] Webserver started at http://127.9.111.65:32933/ using document root <none> and password file <none>
I20260806 17:02:37.216992 13417 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:37.217093 13417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:37.217368 13417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:37.219748 13417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "139a94145ce04a039fa484b195f85e5b"
format_stamp: "Formatted at 2026-08-06 17:02:37 on dist-test-slave-69rg"
server_key: "c5e0fbe8648f6dda2031782c11d56f3c"
server_key_iv: "2c26191fd0b382ef44c9e76af0ca4f8d"
server_key_version: "encryptionkey@0"
I20260806 17:02:37.220402 13417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "139a94145ce04a039fa484b195f85e5b"
format_stamp: "Formatted at 2026-08-06 17:02:37 on dist-test-slave-69rg"
server_key: "c5e0fbe8648f6dda2031782c11d56f3c"
server_key_iv: "2c26191fd0b382ef44c9e76af0ca4f8d"
server_key_version: "encryptionkey@0"
I20260806 17:02:37.224560 13417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260806 17:02:37.227448 13433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:37.228820 13417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 17:02:37.228984 13417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "139a94145ce04a039fa484b195f85e5b"
format_stamp: "Formatted at 2026-08-06 17:02:37 on dist-test-slave-69rg"
server_key: "c5e0fbe8648f6dda2031782c11d56f3c"
server_key_iv: "2c26191fd0b382ef44c9e76af0ca4f8d"
server_key_version: "encryptionkey@0"
I20260806 17:02:37.229166 13417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:37.245919 13417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:37.249591 13417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:37.249876 13417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:37.250551 13417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:37.251427 13417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:37.251475 13417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:37.251539 13417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:37.251571 13417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:37.261020 13417 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:34617
I20260806 17:02:37.261045 13549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:34617 every 8 connection(s)
I20260806 17:02:37.262121 13417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:02:37.267669 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 13417
I20260806 17:02:37.267849 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:02:37.268177 9661 external_mini_cluster.cc:1428] Setting key efcad1c24ea547f00a1b52063bff4516
Aug 06 17:02:37 dist-test-slave-69rg krb5kdc[13304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:37.279013 13552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:37.264046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:50323 (local address 127.9.111.126:33525)
0806 17:02:37.264339 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:37.264344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:37.265123 (+ 779us) server_negotiation.cc:407] Connection header received
0806 17:02:37.266620 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:37.266623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:37.266673 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:37.266760 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:37.268241 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:37.269008 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:37.270532 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:37.270787 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:37.270904 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:37.273561 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:37.273567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:37.273573 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:37.273609 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:37.276555 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:37.277210 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:37.277214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:37.277216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:37.277326 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:37.277706 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:37.277709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:37.277733 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:37.277994 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:37.278122 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:37.278658 (+ 536us) server_negotiation.cc:299] Negotiation successful
0806 17:02:37.278815 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
I20260806 17:02:37.279950 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:37.264494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:33525 (local address 127.9.111.65:50323)
0806 17:02:37.264959 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:37.265003 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:37.266417 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:37.266932 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:37.266938 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:37.267425 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:37.268063 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:37.268083 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:37.269172 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:37.269175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:37.270390 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:37.270400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:37.270943 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:37.270948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:37.271135 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:37.271786 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:37.271802 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:37.273330 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:37.276714 (+ 3384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:37.276720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:37.276734 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:37.277080 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:37.277434 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:37.277438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:37.277440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:37.277594 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:37.278120 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:37.278128 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:37.278396 (+ 268us) client_negotiation.cc:769] Sending connection context
0806 17:02:37.278599 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 17:02:37.278961 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.9.111.66@KRBTEST.COM; defaulting to no policy
I20260806 17:02:37.281351 13550 heartbeater.cc:347] Connected to a master server at 127.9.111.126:33525
I20260806 17:02:37.281626 13550 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:37.282177 13550 heartbeater.cc:511] Master 127.9.111.126:33525 requested a full tablet report, sending...
I20260806 17:02:37.283819 13351 ts_manager.cc:194] Registered new tserver with Master: 139a94145ce04a039fa484b195f85e5b (127.9.111.65:34617)
I20260806 17:02:37.285050 13351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:50323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
W20260806 17:02:37.468350 9661 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.9.111.66: real 0.198s user 0.004s sys 0.001s
I20260806 17:02:37.468914 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:33525
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:42971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:37.584824 13556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:37.585155 13556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:37.585230 13556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:37.589548 13556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:37.589622 13556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:37.589682 13556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:02:37.595245 13556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:42971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:33525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:37.596469 13556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:37.597537 13556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:37.607414 13561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:37.607406 13564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:37.607450 13562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:37.607861 13556 server_base.cc:1034] running on GCE node
I20260806 17:02:37.608328 13556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:37.609031 13556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:37.610234 13556 hybrid_clock.cc:648] HybridClock initialized: now 1786035757610223 us; error 35 us; skew 500 ppm
Aug 06 17:02:37 dist-test-slave-69rg krb5kdc[13304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:37.613391 13556 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:02:37.614507 13556 webserver.cc:460] Webserver started at http://127.9.111.66:44971/ using document root <none> and password file <none>
I20260806 17:02:37.615061 13556 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:37.615124 13556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:37.615309 13556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:37.617491 13556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "5d85cf253e424e97a94755972c503686"
format_stamp: "Formatted at 2026-08-06 17:02:37 on dist-test-slave-69rg"
server_key: "9cb0fcd6c3048f4fb9f7c6d81b988240"
server_key_iv: "1f4cd07989aa71f091d7bcbd81679cb6"
server_key_version: "encryptionkey@0"
I20260806 17:02:37.618089 13556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "5d85cf253e424e97a94755972c503686"
format_stamp: "Formatted at 2026-08-06 17:02:37 on dist-test-slave-69rg"
server_key: "9cb0fcd6c3048f4fb9f7c6d81b988240"
server_key_iv: "1f4cd07989aa71f091d7bcbd81679cb6"
server_key_version: "encryptionkey@0"
I20260806 17:02:37.621930 13556 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 17:02:37.624570 13571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:37.625975 13556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 17:02:37.626078 13556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "5d85cf253e424e97a94755972c503686"
format_stamp: "Formatted at 2026-08-06 17:02:37 on dist-test-slave-69rg"
server_key: "9cb0fcd6c3048f4fb9f7c6d81b988240"
server_key_iv: "1f4cd07989aa71f091d7bcbd81679cb6"
server_key_version: "encryptionkey@0"
I20260806 17:02:37.626178 13556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:37.652648 13556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:37.656322 13556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:37.656495 13556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:37.657050 13556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:37.657939 13556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:37.657989 13556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:37.658030 13556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:37.658046 13556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:37.666764 13556 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:42151
I20260806 17:02:37.666796 13685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:42151 every 8 connection(s)
I20260806 17:02:37.667711 13556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
Aug 06 17:02:37 dist-test-slave-69rg krb5kdc[13304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:37.677968 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 13556
I20260806 17:02:37.678092 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:02:37.678463 9661 external_mini_cluster.cc:1428] Setting key b69ad6fce92ea56593ddecf231b2a86a
I20260806 17:02:37.683250 13552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:37.669723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:51569 (local address 127.9.111.126:33525)
0806 17:02:37.669937 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:37.669941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:37.670804 (+ 863us) server_negotiation.cc:407] Connection header received
0806 17:02:37.672249 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:37.672252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:37.672310 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:37.672392 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:37.673624 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:37.674183 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:37.675407 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:37.675641 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:37.675724 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:37.678654 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:37.678658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:37.678661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:37.678696 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:37.680488 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:37.681203 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:37.681206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:37.681207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:37.681293 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:37.681743 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:37.681747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:37.681748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:37.682028 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:37.682119 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:37.682816 (+ 697us) server_negotiation.cc:299] Negotiation successful
0806 17:02:37.682984 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 17:02:37.683982 13688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:37.670116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:33525 (local address 127.9.111.66:51569)
0806 17:02:37.670615 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:37.670665 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:37.672018 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:37.672557 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:37.672563 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:37.672934 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:37.673464 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:37.673478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:37.674334 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:37.674336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:37.675269 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:37.675280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:37.675790 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:37.675793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:37.675937 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:37.676670 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:37.676687 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:37.678433 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:37.680627 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:37.680633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:37.680655 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:37.681060 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:37.681392 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:37.681398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:37.681403 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:37.681613 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:37.682140 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:37.682150 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:37.682551 (+ 401us) client_negotiation.cc:769] Sending connection context
0806 17:02:37.682726 (+ 175us) client_negotiation.cc:240] Negotiation successful
0806 17:02:37.683012 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":131,"threads_started":1}
I20260806 17:02:37.685253 13686 heartbeater.cc:347] Connected to a master server at 127.9.111.126:33525
I20260806 17:02:37.685621 13686 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:37.686254 13686 heartbeater.cc:511] Master 127.9.111.126:33525 requested a full tablet report, sending...
I20260806 17:02:37.687467 13351 ts_manager.cc:194] Registered new tserver with Master: 5d85cf253e424e97a94755972c503686 (127.9.111.66:42151)
I20260806 17:02:37.688084 13351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:51569
WARNING: no policy specified for kudu/127.9.111.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
I20260806 17:02:37.714605 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:33525
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:42971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:37.829595 13691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:37.829944 13691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:37.830024 13691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:37.834551 13691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:37.834661 13691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:37.834758 13691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:02:37.840348 13691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:42971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:33525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:37.841710 13691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:37.842859 13691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:37.852802 13699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:37.852775 13697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:37.852775 13696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:37.853229 13691 server_base.cc:1034] running on GCE node
I20260806 17:02:37.853711 13691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:37.854491 13691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:37.855710 13691 hybrid_clock.cc:648] HybridClock initialized: now 1786035757855683 us; error 40 us; skew 500 ppm
Aug 06 17:02:37 dist-test-slave-69rg krb5kdc[13304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:37.858731 13691 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:02:37.859656 13691 webserver.cc:460] Webserver started at http://127.9.111.67:34083/ using document root <none> and password file <none>
I20260806 17:02:37.860251 13691 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:37.860347 13691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:37.860590 13691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:37.862804 13691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "cd10357b380e42c9b9bb3b2d406838a4"
format_stamp: "Formatted at 2026-08-06 17:02:37 on dist-test-slave-69rg"
server_key: "8b394abee89002d01dda995b1334c31a"
server_key_iv: "4628c91da4f30f6015d6f2d01c0020ab"
server_key_version: "encryptionkey@0"
I20260806 17:02:37.863415 13691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "cd10357b380e42c9b9bb3b2d406838a4"
format_stamp: "Formatted at 2026-08-06 17:02:37 on dist-test-slave-69rg"
server_key: "8b394abee89002d01dda995b1334c31a"
server_key_iv: "4628c91da4f30f6015d6f2d01c0020ab"
server_key_version: "encryptionkey@0"
I20260806 17:02:37.867350 13691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:02:37.870038 13707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:37.871275 13691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 17:02:37.871418 13691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "cd10357b380e42c9b9bb3b2d406838a4"
format_stamp: "Formatted at 2026-08-06 17:02:37 on dist-test-slave-69rg"
server_key: "8b394abee89002d01dda995b1334c31a"
server_key_iv: "4628c91da4f30f6015d6f2d01c0020ab"
server_key_version: "encryptionkey@0"
I20260806 17:02:37.871534 13691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:37.883217 13691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:37.886642 13691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:37.886862 13691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:37.887480 13691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:37.888319 13691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:37.888399 13691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:37.888474 13691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:37.888517 13691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:37.897989 13691 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:39667
I20260806 17:02:37.898064 13822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:39667 every 8 connection(s)
I20260806 17:02:37.898969 13691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:02:37.902604 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 13691
I20260806 17:02:37.902799 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:02:37.903136 9661 external_mini_cluster.cc:1428] Setting key a1136094c2ba28fa37f0b371391ee930
Aug 06 17:02:37 dist-test-slave-69rg krb5kdc[13304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:37.913316 13552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:37.900877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:50063 (local address 127.9.111.126:33525)
0806 17:02:37.901048 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:37.901052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:37.901887 (+ 835us) server_negotiation.cc:407] Connection header received
0806 17:02:37.903782 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:37.903786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:37.903884 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:37.903976 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:37.905528 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:37.906149 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:37.907265 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:37.907483 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:37.907560 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:37.910103 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:37.910106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:37.910109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:37.910133 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:37.911400 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:37.911915 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:37.911918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:37.911919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:37.911993 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:37.912358 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:37.912360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:37.912361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:37.912532 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:37.912607 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:37.913096 (+ 489us) server_negotiation.cc:299] Negotiation successful
0806 17:02:37.913193 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 17:02:37.914250 13825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:37.901229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:33525 (local address 127.9.111.67:50063)
0806 17:02:37.901704 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:37.901747 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:37.903529 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:37.904209 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:37.904218 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:37.904768 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:37.905382 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:37.905401 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:37.906261 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:37.906264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:37.907145 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:37.907154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:37.907611 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:37.907613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:37.907754 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:37.908395 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:37.908412 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:37.909953 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:37.911517 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:37.911522 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:37.911534 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:37.911827 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:37.912104 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:37.912107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:37.912109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:37.912267 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:37.912640 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:37.912645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:37.912888 (+ 243us) client_negotiation.cc:769] Sending connection context
0806 17:02:37.913046 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 17:02:37.913313 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":128,"threads_started":1}
I20260806 17:02:37.915465 13823 heartbeater.cc:347] Connected to a master server at 127.9.111.126:33525
I20260806 17:02:37.915724 13823 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:37.916314 13823 heartbeater.cc:511] Master 127.9.111.126:33525 requested a full tablet report, sending...
I20260806 17:02:37.917490 13351 ts_manager.cc:194] Registered new tserver with Master: cd10357b380e42c9b9bb3b2d406838a4 (127.9.111.67:39667)
I20260806 17:02:37.918161 13351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:50063
I20260806 17:02:37.928889 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 17:02:37 dist-test-slave-69rg krb5kdc[13304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 17:02:37 dist-test-slave-69rg krb5kdc[13304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.65@KRBTEST.COM
I20260806 17:02:37.956279 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:37.946594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.9.111.65:34617)
0806 17:02:37.946944 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:37.946950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:37.946975 (+ 25us) server_negotiation.cc:407] Connection header received
0806 17:02:37.947063 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:37.947068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:37.947269 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:37.947393 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:37.948159 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:37.948933 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:37.949895 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:37.950284 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:37.950412 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:37.952929 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:37.952938 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:37.952952 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:37.952976 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:37.954578 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:37.954983 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:37.954988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:37.954989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:37.955068 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:37.955318 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:37.955320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:37.955321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:37.955559 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:37.955710 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:37.955910 (+ 200us) server_negotiation.cc:299] Negotiation successful
0806 17:02:37.956071 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":135,"threads_started":1}
Aug 06 17:02:37 dist-test-slave-69rg krb5kdc[13304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:37.967826 13552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:37.959317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37434 (local address 127.9.111.126:33525)
0806 17:02:37.959454 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:37.959459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:37.959474 (+ 15us) server_negotiation.cc:407] Connection header received
0806 17:02:37.959550 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:37.959555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:37.959601 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:37.959688 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:37.960350 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:37.960866 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:37.961664 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:37.961943 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:37.962050 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:37.964458 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:37.964468 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:37.964469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:37.964497 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:37.966049 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:37.966511 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:37.966515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:37.966516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:37.966593 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:37.966899 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:37.966903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:37.966905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:37.967140 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:37.967296 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:37.967529 (+ 233us) server_negotiation.cc:299] Negotiation successful
0806 17:02:37.967654 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 17:02:37.968281 13351 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37434
I20260806 17:02:37.968972 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 13417
I20260806 17:02:37.976938 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 13556
I20260806 17:02:37.983693 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 13691
I20260806 17:02:37.989538 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 13318
2026-08-06T17:02:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3668 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:02:38 dist-test-slave-69rg krb5kdc[13835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:02:38 dist-test-slave-69rg krb5kdc[13835](info): set up 2 sockets
Aug 06 17:02:38 dist-test-slave-69rg krb5kdc[13835](info): commencing operation
krb5kdc: starting...
W20260806 17:02:39.073642 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:02:39 dist-test-slave-69rg krb5kdc[13835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:02:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:02:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:02:39.218402 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:41091
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:35821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:41091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:39.333586 13848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:39.333993 13848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:39.334084 13848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:39.338290 13848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:39.338407 13848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:39.338442 13848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:39.338485 13848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:39.338519 13848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:02:39.343937 13848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:35821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:41091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:41091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:39.345157 13848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:39.346262 13848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:39.354490 13853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:39.354547 13848 server_base.cc:1034] running on GCE node
W20260806 17:02:39.354499 13856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:39.354493 13854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:39.355427 13848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:39.356545 13848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:39.357720 13848 hybrid_clock.cc:648] HybridClock initialized: now 1786035759357703 us; error 32 us; skew 500 ppm
Aug 06 17:02:39 dist-test-slave-69rg krb5kdc[13835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035759, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:39.364491 13848 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:02:39.365216 13848 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:39.365280 13848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:39.365522 13848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:39.367646 13848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "92ab87e6e42247b9be0e6906fd2fbdc9"
format_stamp: "Formatted at 2026-08-06 17:02:39 on dist-test-slave-69rg"
server_key: "d0c96e0d8f7968f4deb5d30dff8ad53c"
server_key_iv: "1b4e106b96aa72a4320002a96c022d64"
server_key_version: "encryptionkey@0"
I20260806 17:02:39.368217 13848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "92ab87e6e42247b9be0e6906fd2fbdc9"
format_stamp: "Formatted at 2026-08-06 17:02:39 on dist-test-slave-69rg"
server_key: "d0c96e0d8f7968f4deb5d30dff8ad53c"
server_key_iv: "1b4e106b96aa72a4320002a96c022d64"
server_key_version: "encryptionkey@0"
I20260806 17:02:39.372016 13848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 17:02:39.374742 13862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:39.376096 13848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:39.376226 13848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "92ab87e6e42247b9be0e6906fd2fbdc9"
format_stamp: "Formatted at 2026-08-06 17:02:39 on dist-test-slave-69rg"
server_key: "d0c96e0d8f7968f4deb5d30dff8ad53c"
server_key_iv: "1b4e106b96aa72a4320002a96c022d64"
server_key_version: "encryptionkey@0"
I20260806 17:02:39.376345 13848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:39.398918 13848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:39.402405 13848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:39.402626 13848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:39.410080 13848 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:41091
I20260806 17:02:39.410406 13914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:41091 every 8 connection(s)
I20260806 17:02:39.410799 13848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:02:39.413582 13915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:39.416391 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 13848
I20260806 17:02:39.416503 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:02:39.416832 9661 external_mini_cluster.cc:1428] Setting key fae34427a55342def49ff927d5a0ff16
I20260806 17:02:39.419771 13915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9: Bootstrap starting.
I20260806 17:02:39.422546 13915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:39.423452 13915 log.cc:824] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:39.425391 13915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9: No bootstrap required, opened a new log
Aug 06 17:02:39 dist-test-slave-69rg krb5kdc[13835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:39.428947 13915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ab87e6e42247b9be0e6906fd2fbdc9" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41091 } }
I20260806 17:02:39.429180 13915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:39.429224 13915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ab87e6e42247b9be0e6906fd2fbdc9, State: Initialized, Role: FOLLOWER
I20260806 17:02:39.429750 13915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ab87e6e42247b9be0e6906fd2fbdc9" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41091 } }
I20260806 17:02:39.429966 13915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:02:39.430058 13915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:02:39.430182 13915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:39.431468 13915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ab87e6e42247b9be0e6906fd2fbdc9" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41091 } }
I20260806 17:02:39.431844 13915 leader_election.cc:302] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92ab87e6e42247b9be0e6906fd2fbdc9; no voters:
I20260806 17:02:39.432209 13915 leader_election.cc:288] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:02:39.432353 13920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:39.432585 13920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9 [term 1 LEADER]: Becoming Leader. State: Replica: 92ab87e6e42247b9be0e6906fd2fbdc9, State: Running, Role: LEADER
I20260806 17:02:39.432978 13920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ab87e6e42247b9be0e6906fd2fbdc9" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41091 } }
I20260806 17:02:39.433390 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:39.418298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56190 (local address 127.9.111.126:41091)
0806 17:02:39.418758 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:39.418773 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:39.418809 (+ 36us) server_negotiation.cc:407] Connection header received
0806 17:02:39.419416 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:39.419433 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:39.419746 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:39.420065 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:39.420984 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:39.422048 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:39.423254 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:39.423782 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:39.423972 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:39.426800 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:39.426806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:39.426821 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:39.426853 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:39.429603 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:39.430241 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:39.430249 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:39.430254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:39.430431 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:39.430736 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:39.430739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:39.430741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:39.431321 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:39.431547 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:39.431852 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 17:02:39.432225 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":134,"threads_started":1}
I20260806 17:02:39.433542 13915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:39.434340 13920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92ab87e6e42247b9be0e6906fd2fbdc9. Latest consensus state: current_term: 1 leader_uuid: "92ab87e6e42247b9be0e6906fd2fbdc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ab87e6e42247b9be0e6906fd2fbdc9" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41091 } } }
I20260806 17:02:39.434474 13920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:39.434955 13921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92ab87e6e42247b9be0e6906fd2fbdc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ab87e6e42247b9be0e6906fd2fbdc9" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41091 } } }
I20260806 17:02:39.435135 13921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:39.436236 13928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:39.439746 13928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:02:39.445864 13928 catalog_manager.cc:1347] Generated new cluster ID: 90e114cd4e37462bb6a8488329b299b4
I20260806 17:02:39.445946 13928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:39.455955 13928 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:02:39.457244 13928 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:39.462499 13928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9: Generated new TSK 0
I20260806 17:02:39.463300 13928 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
I20260806 17:02:39.504966 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:41091
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:35821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:39.621407 13941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:39.621663 13941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:39.621749 13941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:39.625743 13941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:39.625809 13941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:39.625921 13941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:02:39.631183 13941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:35821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:41091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:39.632256 13941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:39.633250 13941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:39.642992 13950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:39.643009 13947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:39.643019 13948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:39.643424 13941 server_base.cc:1034] running on GCE node
I20260806 17:02:39.643821 13941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:39.644425 13941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:39.645594 13941 hybrid_clock.cc:648] HybridClock initialized: now 1786035759645565 us; error 41 us; skew 500 ppm
Aug 06 17:02:39 dist-test-slave-69rg krb5kdc[13835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035759, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:39.648659 13941 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:02:39.649387 13941 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:39.649453 13941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:39.649624 13941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:39.651751 13941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "72835eb12bf04508876fc3f19eef9dcf"
format_stamp: "Formatted at 2026-08-06 17:02:39 on dist-test-slave-69rg"
server_key: "d2eff5b244f28f0c2e4ecba804752457"
server_key_iv: "51ef1153aac099d0c2fa5ba09b2f3775"
server_key_version: "encryptionkey@0"
I20260806 17:02:39.652266 13941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "72835eb12bf04508876fc3f19eef9dcf"
format_stamp: "Formatted at 2026-08-06 17:02:39 on dist-test-slave-69rg"
server_key: "d2eff5b244f28f0c2e4ecba804752457"
server_key_iv: "51ef1153aac099d0c2fa5ba09b2f3775"
server_key_version: "encryptionkey@0"
I20260806 17:02:39.656193 13941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 17:02:39.658696 13956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:39.659956 13941 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:02:39.660060 13941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "72835eb12bf04508876fc3f19eef9dcf"
format_stamp: "Formatted at 2026-08-06 17:02:39 on dist-test-slave-69rg"
server_key: "d2eff5b244f28f0c2e4ecba804752457"
server_key_iv: "51ef1153aac099d0c2fa5ba09b2f3775"
server_key_version: "encryptionkey@0"
I20260806 17:02:39.660145 13941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:39.678875 13941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:39.682214 13941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:39.682426 13941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:39.683002 13941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:39.683857 13941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:39.683903 13941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:39.683991 13941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:39.684021 13941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:39.692524 13941 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:33813
I20260806 17:02:39.692651 14073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:33813 every 8 connection(s)
I20260806 17:02:39.693156 13941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:02:39.702814 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 13941
I20260806 17:02:39.702961 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
Aug 06 17:02:39 dist-test-slave-69rg krb5kdc[13835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035759, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:39.703293 9661 external_mini_cluster.cc:1428] Setting key f8c5df986ed8a5260464e1822e5f0e7d
I20260806 17:02:39.708231 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:39.694985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:38235 (local address 127.9.111.126:41091)
0806 17:02:39.695136 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:39.695140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:39.696339 (+ 1199us) server_negotiation.cc:407] Connection header received
0806 17:02:39.697246 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:39.697249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:39.697311 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:39.697426 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:39.698743 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:39.699300 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:39.700469 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:39.700699 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:39.700817 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:39.703869 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:39.703874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:39.703877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:39.703911 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:39.705648 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:39.706353 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:39.706356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:39.706357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:39.706437 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:39.706926 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:39.706928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:39.706929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:39.707114 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:39.707194 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:39.707887 (+ 693us) server_negotiation.cc:299] Negotiation successful
0806 17:02:39.708041 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 17:02:39.709187 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:39.695358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:41091 (local address 127.9.111.65:38235)
0806 17:02:39.696172 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:39.696216 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:39.697036 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:39.697631 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:39.697637 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:39.698086 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:39.698590 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:39.698602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:39.699435 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:39.699445 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:39.700326 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:39.700335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:39.700870 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:39.700874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:39.701048 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:39.701670 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:39.701687 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:39.703606 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:39.705790 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:39.705809 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:39.705849 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:39.706250 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:39.706567 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:39.706572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:39.706577 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:39.706794 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:39.707219 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:39.707229 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:39.707604 (+ 375us) client_negotiation.cc:769] Sending connection context
0806 17:02:39.707784 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 17:02:39.708072 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":115,"threads_started":1}
I20260806 17:02:39.710530 14074 heartbeater.cc:347] Connected to a master server at 127.9.111.126:41091
I20260806 17:02:39.710799 14074 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:39.711277 14074 heartbeater.cc:511] Master 127.9.111.126:41091 requested a full tablet report, sending...
I20260806 17:02:39.712898 13879 ts_manager.cc:194] Registered new tserver with Master: 72835eb12bf04508876fc3f19eef9dcf (127.9.111.65:33813)
I20260806 17:02:39.714125 13879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:38235
WARNING: no policy specified for kudu/127.9.111.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
I20260806 17:02:39.733541 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:41091
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:35821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:39.845093 14079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:39.845404 14079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:39.845485 14079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:39.849462 14079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:39.849560 14079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:39.849655 14079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:02:39.854871 14079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:35821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:41091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:39.855952 14079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:39.856961 14079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:39.866305 14085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:39.866348 14084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:39.866346 14087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:39.866385 14079 server_base.cc:1034] running on GCE node
I20260806 17:02:39.866915 14079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:39.867604 14079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:39.868759 14079 hybrid_clock.cc:648] HybridClock initialized: now 1786035759868744 us; error 32 us; skew 500 ppm
Aug 06 17:02:39 dist-test-slave-69rg krb5kdc[13835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035759, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:39.871589 14079 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:02:39.872278 14079 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:39.872347 14079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:39.872507 14079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:39.874574 14079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "840e24db2484400389c16071b2471753"
format_stamp: "Formatted at 2026-08-06 17:02:39 on dist-test-slave-69rg"
server_key: "2a233a3dbc2b252752ad3413c34733a8"
server_key_iv: "fe56554130bad4e721ee5a0aac5e59cb"
server_key_version: "encryptionkey@0"
I20260806 17:02:39.875077 14079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "840e24db2484400389c16071b2471753"
format_stamp: "Formatted at 2026-08-06 17:02:39 on dist-test-slave-69rg"
server_key: "2a233a3dbc2b252752ad3413c34733a8"
server_key_iv: "fe56554130bad4e721ee5a0aac5e59cb"
server_key_version: "encryptionkey@0"
I20260806 17:02:39.878535 14079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 17:02:39.881237 14093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:39.882530 14079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:02:39.882645 14079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "840e24db2484400389c16071b2471753"
format_stamp: "Formatted at 2026-08-06 17:02:39 on dist-test-slave-69rg"
server_key: "2a233a3dbc2b252752ad3413c34733a8"
server_key_iv: "fe56554130bad4e721ee5a0aac5e59cb"
server_key_version: "encryptionkey@0"
I20260806 17:02:39.882755 14079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:39.900348 14079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:39.903911 14079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:39.904070 14079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:39.904616 14079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:39.905530 14079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:39.905580 14079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:39.905619 14079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:39.905635 14079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:39.915016 14079 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:32811
I20260806 17:02:39.915037 14207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:32811 every 8 connection(s)
I20260806 17:02:39.915714 14079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:02:39.921131 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 14079
I20260806 17:02:39.921280 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:02:39.921564 9661 external_mini_cluster.cc:1428] Setting key 0009101796010f0d78871e39e96d1982
Aug 06 17:02:39 dist-test-slave-69rg krb5kdc[13835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035759, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:39.930894 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:39.917607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:55401 (local address 127.9.111.126:41091)
0806 17:02:39.917761 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:39.917765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:39.919012 (+ 1247us) server_negotiation.cc:407] Connection header received
0806 17:02:39.919852 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:39.919855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:39.919905 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:39.919984 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:39.921273 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:39.921972 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:39.923415 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:39.923630 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:39.923720 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:39.926703 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:39.926709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:39.926715 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:39.926752 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:39.928498 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:39.929027 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:39.929032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:39.929034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:39.929167 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:39.929516 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:39.929522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:39.929526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:39.929871 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:39.930038 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:39.930535 (+ 497us) server_negotiation.cc:299] Negotiation successful
0806 17:02:39.930687 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 17:02:39.931633 14210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:39.918054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:41091 (local address 127.9.111.66:55401)
0806 17:02:39.918861 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:39.918901 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:39.919655 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:39.920141 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:39.920147 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:39.920509 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:39.921077 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:39.921097 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:39.922132 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:39.922135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:39.923278 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:39.923290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:39.923798 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:39.923803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:39.924030 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:39.924724 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:39.924750 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:39.926525 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:39.928631 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:39.928637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:39.928650 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:39.928917 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:39.929266 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:39.929268 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:39.929270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:39.929417 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:39.930019 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:39.930025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:39.930267 (+ 242us) client_negotiation.cc:769] Sending connection context
0806 17:02:39.930443 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 17:02:39.930706 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.9.111.67@KRBTEST.COM; defaulting to no policy
I20260806 17:02:39.932914 14208 heartbeater.cc:347] Connected to a master server at 127.9.111.126:41091
I20260806 17:02:39.933151 14208 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:39.933647 14208 heartbeater.cc:511] Master 127.9.111.126:41091 requested a full tablet report, sending...
I20260806 17:02:39.934900 13879 ts_manager.cc:194] Registered new tserver with Master: 840e24db2484400389c16071b2471753 (127.9.111.66:32811)
I20260806 17:02:39.935540 13879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:55401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
I20260806 17:02:39.952980 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:41091
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:35821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:40.073342 14213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:40.073709 14213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:40.073797 14213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:40.077999 14213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:40.078083 14213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:40.078213 14213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:02:40.083932 14213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:35821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:41091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:40.085124 14213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:40.086380 14213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:40.095959 14219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:40.096133 14213 server_base.cc:1034] running on GCE node
W20260806 17:02:40.095966 14218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:40.095966 14221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:40.096684 14213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:40.097265 14213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:40.098436 14213 hybrid_clock.cc:648] HybridClock initialized: now 1786035760098423 us; error 24 us; skew 500 ppm
Aug 06 17:02:40 dist-test-slave-69rg krb5kdc[13835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035760, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:40.101334 14213 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:02:40.102101 14213 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:40.102167 14213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:40.102341 14213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:40.104341 14213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "562617eb5bc74263939493c82d6467ff"
format_stamp: "Formatted at 2026-08-06 17:02:40 on dist-test-slave-69rg"
server_key: "b08d7a7b04d5a7f0531a29fb446d32f2"
server_key_iv: "3180da7302a0738caa7f20cd9669af75"
server_key_version: "encryptionkey@0"
I20260806 17:02:40.104861 14213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "562617eb5bc74263939493c82d6467ff"
format_stamp: "Formatted at 2026-08-06 17:02:40 on dist-test-slave-69rg"
server_key: "b08d7a7b04d5a7f0531a29fb446d32f2"
server_key_iv: "3180da7302a0738caa7f20cd9669af75"
server_key_version: "encryptionkey@0"
I20260806 17:02:40.108454 14213 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 17:02:40.110867 14227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:40.112083 14213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:02:40.112275 14213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "562617eb5bc74263939493c82d6467ff"
format_stamp: "Formatted at 2026-08-06 17:02:40 on dist-test-slave-69rg"
server_key: "b08d7a7b04d5a7f0531a29fb446d32f2"
server_key_iv: "3180da7302a0738caa7f20cd9669af75"
server_key_version: "encryptionkey@0"
I20260806 17:02:40.112392 14213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:40.134685 14213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:40.137795 14213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:40.138052 14213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:40.138684 14213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:40.139528 14213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:40.139600 14213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:40.139711 14213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:40.139765 14213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:40.148846 14213 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:45689
I20260806 17:02:40.148870 14344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:45689 every 8 connection(s)
I20260806 17:02:40.149462 14213 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:02:40.150672 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 14213
I20260806 17:02:40.150830 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:02:40.151199 9661 external_mini_cluster.cc:1428] Setting key 9aa750512eff8dda793003d16e4718d8
Aug 06 17:02:40 dist-test-slave-69rg krb5kdc[13835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035760, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:40.163542 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.151352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:35419 (local address 127.9.111.126:41091)
0806 17:02:40.151506 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:40.151511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:40.152474 (+ 963us) server_negotiation.cc:407] Connection header received
0806 17:02:40.153976 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:40.153978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:40.154027 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:40.154097 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:40.155300 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.155864 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.157029 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.157255 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.157344 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:40.159928 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:40.159931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:40.159934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:40.159962 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:40.161548 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:40.162105 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:40.162108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:40.162109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:40.162196 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:40.162492 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:40.162494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:40.162495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:40.162674 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:40.162749 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:40.163253 (+ 504us) server_negotiation.cc:299] Negotiation successful
0806 17:02:40.163385 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 17:02:40.164702 14347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.151790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:41091 (local address 127.9.111.67:35419)
0806 17:02:40.152326 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:40.152380 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:40.153723 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:40.154301 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:40.154307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:40.154660 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:40.155155 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:40.155169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.156000 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.156003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:40.156901 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:40.156911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.157410 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.157412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:40.157557 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:40.158178 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:40.158200 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:40.159745 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:40.161653 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:40.161657 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:40.161670 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:40.162017 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:40.162281 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:40.162283 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:40.162284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:40.162428 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:40.162758 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:40.162763 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:40.163017 (+ 254us) client_negotiation.cc:769] Sending connection context
0806 17:02:40.163172 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 17:02:40.163510 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":157,"threads_started":1}
I20260806 17:02:40.166090 14345 heartbeater.cc:347] Connected to a master server at 127.9.111.126:41091
I20260806 17:02:40.166402 14345 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:40.166978 14345 heartbeater.cc:511] Master 127.9.111.126:41091 requested a full tablet report, sending...
I20260806 17:02:40.168329 13879 ts_manager.cc:194] Registered new tserver with Master: 562617eb5bc74263939493c82d6467ff (127.9.111.67:45689)
I20260806 17:02:40.169063 13879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:35419
I20260806 17:02:40.177898 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 17:02:40.187597 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.179985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56200 (local address 127.9.111.126:41091)
0806 17:02:40.180139 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:40.180149 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:40.180320 (+ 171us) server_negotiation.cc:407] Connection header received
0806 17:02:40.180428 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:40.180430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:40.180476 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:40.180541 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:40.181425 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.181970 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.182775 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.182985 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.183074 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:40.184370 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:40.184373 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:40.184375 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:40.184399 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:40.185897 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:40.186278 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:40.186281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:40.186301 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:40.186383 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:40.186619 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:40.186621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:40.186622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:40.186856 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:40.186987 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:40.187252 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 17:02:40.187397 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 17:02:40.190688 13879 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:02:40.193775 13879 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:02:40.208207 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.202499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:45689 (local address 127.0.0.1:39172)
0806 17:02:40.203431 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:40.203447 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:40.203536 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:40.204166 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:40.204168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:40.204182 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:40.204291 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:40.204297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.205327 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.205336 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:40.206761 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:40.206773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.207648 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.207651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:40.207756 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:40.207793 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 17:02:40.207922 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 17:02:40.208045 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"thread_start_us":75,"threads_started":1}
I20260806 17:02:40.208403 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.202226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:33813 (local address 127.0.0.1:40270)
0806 17:02:40.203027 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:40.203080 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:40.203207 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:40.204218 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:40.204224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:40.204247 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:40.204346 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:40.204351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.205740 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.205743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:40.206938 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:40.206948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.207945 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.207948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:40.208026 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:40.208045 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 17:02:40.208101 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 17:02:40.208157 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":50,"thread_start_us":114,"threads_started":1}
I20260806 17:02:40.208709 14356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.202134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:32811 (local address 127.0.0.1:41438)
0806 17:02:40.203028 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:40.203077 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:40.203211 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:40.203991 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:40.203999 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:40.204025 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:40.204176 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:40.204187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.205676 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.205679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:40.206892 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:40.206899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.208261 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.208265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:40.208378 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:40.208403 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 17:02:40.208474 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 17:02:40.208559 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":25,"thread_start_us":79,"threads_started":1}
I20260806 17:02:40.208776 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.203374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.9.111.65:33813)
0806 17:02:40.203634 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:40.203640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:40.203664 (+ 24us) server_negotiation.cc:407] Connection header received
0806 17:02:40.203739 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:40.203743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:40.203916 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:40.204060 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:40.204498 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.205607 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.207067 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.207828 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.207938 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:40.208065 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:40.208129 (+ 64us) server_negotiation.cc:299] Negotiation successful
0806 17:02:40.208213 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":80,"threads_started":1}
I20260806 17:02:40.208920 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.202254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41438 (local address 127.9.111.66:32811)
0806 17:02:40.202840 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:40.202857 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:40.203180 (+ 323us) server_negotiation.cc:407] Connection header received
0806 17:02:40.203390 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:40.203396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:40.203607 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:40.203739 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:40.204314 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.205528 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.207002 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.207828 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.207939 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:40.208065 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:40.208652 (+ 587us) server_negotiation.cc:299] Negotiation successful
0806 17:02:40.208759 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":130,"threads_started":1}
I20260806 17:02:40.209158 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.202931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39172 (local address 127.9.111.67:45689)
0806 17:02:40.203414 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:40.203426 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:40.203626 (+ 200us) server_negotiation.cc:407] Connection header received
0806 17:02:40.203705 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:40.203710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:40.203888 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:40.204028 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:40.204429 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.205187 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.206895 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.207521 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.207645 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:40.207798 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:40.208893 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0806 17:02:40.209004 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":183,"threads_started":1}
I20260806 17:02:40.211838 14276 tablet_service.cc:1467] Processing CreateTablet for tablet fce33b2cacb143c0b5204c6b5c8e154e (DEFAULT_TABLE table=test-table [id=6d8efb3da32c431291e2f274d3f4d079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:40.211864 14007 tablet_service.cc:1467] Processing CreateTablet for tablet fce33b2cacb143c0b5204c6b5c8e154e (DEFAULT_TABLE table=test-table [id=6d8efb3da32c431291e2f274d3f4d079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:40.211864 14142 tablet_service.cc:1467] Processing CreateTablet for tablet fce33b2cacb143c0b5204c6b5c8e154e (DEFAULT_TABLE table=test-table [id=6d8efb3da32c431291e2f274d3f4d079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:40.212880 14142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce33b2cacb143c0b5204c6b5c8e154e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:40.212880 14007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce33b2cacb143c0b5204c6b5c8e154e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:40.212883 14276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce33b2cacb143c0b5204c6b5c8e154e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:40.219391 14362 tablet_bootstrap.cc:492] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff: Bootstrap starting.
I20260806 17:02:40.220834 14364 tablet_bootstrap.cc:492] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753: Bootstrap starting.
I20260806 17:02:40.221908 14362 tablet_bootstrap.cc:654] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:40.222640 14364 tablet_bootstrap.cc:654] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:40.222983 14362 log.cc:824] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:40.223680 14363 tablet_bootstrap.cc:492] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf: Bootstrap starting.
I20260806 17:02:40.223692 14364 log.cc:824] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:40.225179 14362 tablet_bootstrap.cc:492] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff: No bootstrap required, opened a new log
I20260806 17:02:40.225400 14362 ts_tablet_manager.cc:1397] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260806 17:02:40.225544 14364 tablet_bootstrap.cc:492] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753: No bootstrap required, opened a new log
I20260806 17:02:40.225649 14363 tablet_bootstrap.cc:654] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:40.225723 14364 ts_tablet_manager.cc:1397] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 17:02:40.226354 14363 log.cc:824] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:40.227911 14363 tablet_bootstrap.cc:492] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf: No bootstrap required, opened a new log
I20260806 17:02:40.228106 14363 ts_tablet_manager.cc:1397] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 17:02:40.229451 14362 raft_consensus.cc:348] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840e24db2484400389c16071b2471753" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 32811 } } peers { permanent_uuid: "72835eb12bf04508876fc3f19eef9dcf" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33813 } } peers { permanent_uuid: "562617eb5bc74263939493c82d6467ff" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 45689 } }
I20260806 17:02:40.229449 14364 raft_consensus.cc:348] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840e24db2484400389c16071b2471753" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 32811 } } peers { permanent_uuid: "72835eb12bf04508876fc3f19eef9dcf" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33813 } } peers { permanent_uuid: "562617eb5bc74263939493c82d6467ff" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 45689 } }
I20260806 17:02:40.229753 14362 raft_consensus.cc:374] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:40.229753 14364 raft_consensus.cc:374] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:40.229837 14362 raft_consensus.cc:729] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562617eb5bc74263939493c82d6467ff, State: Initialized, Role: FOLLOWER
I20260806 17:02:40.229852 14364 raft_consensus.cc:729] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840e24db2484400389c16071b2471753, State: Initialized, Role: FOLLOWER
I20260806 17:02:40.230329 14362 consensus_queue.cc:260] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840e24db2484400389c16071b2471753" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 32811 } } peers { permanent_uuid: "72835eb12bf04508876fc3f19eef9dcf" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33813 } } peers { permanent_uuid: "562617eb5bc74263939493c82d6467ff" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 45689 } }
I20260806 17:02:40.230329 14364 consensus_queue.cc:260] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840e24db2484400389c16071b2471753" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 32811 } } peers { permanent_uuid: "72835eb12bf04508876fc3f19eef9dcf" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33813 } } peers { permanent_uuid: "562617eb5bc74263939493c82d6467ff" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 45689 } }
I20260806 17:02:40.231388 14364 ts_tablet_manager.cc:1428] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 17:02:40.231388 14362 ts_tablet_manager.cc:1428] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 17:02:40.231750 14345 heartbeater.cc:503] Master 127.9.111.126:41091 was elected leader, sending a full tablet report...
I20260806 17:02:40.231751 14208 heartbeater.cc:503] Master 127.9.111.126:41091 was elected leader, sending a full tablet report...
I20260806 17:02:40.232503 14363 raft_consensus.cc:348] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840e24db2484400389c16071b2471753" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 32811 } } peers { permanent_uuid: "72835eb12bf04508876fc3f19eef9dcf" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33813 } } peers { permanent_uuid: "562617eb5bc74263939493c82d6467ff" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 45689 } }
I20260806 17:02:40.232746 14363 raft_consensus.cc:374] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:40.232816 14363 raft_consensus.cc:729] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72835eb12bf04508876fc3f19eef9dcf, State: Initialized, Role: FOLLOWER
I20260806 17:02:40.233242 14363 consensus_queue.cc:260] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840e24db2484400389c16071b2471753" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 32811 } } peers { permanent_uuid: "72835eb12bf04508876fc3f19eef9dcf" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33813 } } peers { permanent_uuid: "562617eb5bc74263939493c82d6467ff" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 45689 } }
I20260806 17:02:40.233927 14074 heartbeater.cc:503] Master 127.9.111.126:41091 was elected leader, sending a full tablet report...
I20260806 17:02:40.234284 14363 ts_tablet_manager.cc:1428] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260806 17:02:40.256994 14370 raft_consensus.cc:482] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:40.257236 14370 raft_consensus.cc:504] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840e24db2484400389c16071b2471753" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 32811 } } peers { permanent_uuid: "72835eb12bf04508876fc3f19eef9dcf" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33813 } } peers { permanent_uuid: "562617eb5bc74263939493c82d6467ff" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 45689 } }
I20260806 17:02:40.258544 14370 leader_election.cc:288] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 840e24db2484400389c16071b2471753 (127.9.111.66:32811), 562617eb5bc74263939493c82d6467ff (127.9.111.67:45689)
I20260806 17:02:40.262423 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.258907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:39171 (local address 127.9.111.66:32811)
0806 17:02:40.259045 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:40.259049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:40.259249 (+ 200us) server_negotiation.cc:407] Connection header received
0806 17:02:40.259392 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:40.259395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:40.259436 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:40.259520 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:40.259827 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.260331 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.261394 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.261791 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.261969 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:40.262021 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:40.262229 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 17:02:40.262294 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 17:02:40.262449 14371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.258752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:32811 (local address 127.9.111.65:39171)
0806 17:02:40.259129 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:40.259144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:40.259275 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:40.259530 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:40.259533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:40.259548 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:40.259710 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:40.259716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.260440 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.260445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:40.261258 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:40.261266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.261961 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.261964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:40.262085 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:40.262124 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 17:02:40.262212 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 17:02:40.262285 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":44,"thread_start_us":136,"threads_started":1}
I20260806 17:02:40.262523 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.258945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:48479 (local address 127.9.111.67:45689)
0806 17:02:40.259153 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:40.259159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:40.259295 (+ 136us) server_negotiation.cc:407] Connection header received
0806 17:02:40.259452 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:40.259460 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:40.259503 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:40.259586 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:40.259904 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.260409 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.261453 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.261925 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.262055 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:40.262115 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:40.262323 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 17:02:40.262392 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260806 17:02:40.263047 14372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.258816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:45689 (local address 127.9.111.65:48479)
0806 17:02:40.259227 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:40.259239 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:40.259324 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:40.259599 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:40.259601 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:40.259617 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:40.259778 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:40.259791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.260527 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.260532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:40.261323 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:40.261332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.262048 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.262052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:40.262198 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:40.262216 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 17:02:40.262306 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 17:02:40.262372 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":44,"thread_start_us":117,"threads_started":1}
I20260806 17:02:40.263316 14162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce33b2cacb143c0b5204c6b5c8e154e" candidate_uuid: "72835eb12bf04508876fc3f19eef9dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "840e24db2484400389c16071b2471753" is_pre_election: true
I20260806 17:02:40.263685 14162 raft_consensus.cc:2454] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72835eb12bf04508876fc3f19eef9dcf in term 0.
I20260806 17:02:40.263830 14297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce33b2cacb143c0b5204c6b5c8e154e" candidate_uuid: "72835eb12bf04508876fc3f19eef9dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562617eb5bc74263939493c82d6467ff" is_pre_election: true
I20260806 17:02:40.264146 14297 raft_consensus.cc:2454] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72835eb12bf04508876fc3f19eef9dcf in term 0.
I20260806 17:02:40.264192 13957 leader_election.cc:302] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72835eb12bf04508876fc3f19eef9dcf, 840e24db2484400389c16071b2471753; no voters:
I20260806 17:02:40.264525 14370 raft_consensus.cc:2781] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:02:40.264636 14370 raft_consensus.cc:482] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:02:40.264695 14370 raft_consensus.cc:3037] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:40.265856 14370 raft_consensus.cc:504] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840e24db2484400389c16071b2471753" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 32811 } } peers { permanent_uuid: "72835eb12bf04508876fc3f19eef9dcf" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33813 } } peers { permanent_uuid: "562617eb5bc74263939493c82d6467ff" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 45689 } }
I20260806 17:02:40.266229 14370 leader_election.cc:288] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf [CANDIDATE]: Term 1 election: Requested vote from peers 840e24db2484400389c16071b2471753 (127.9.111.66:32811), 562617eb5bc74263939493c82d6467ff (127.9.111.67:45689)
I20260806 17:02:40.266605 14162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce33b2cacb143c0b5204c6b5c8e154e" candidate_uuid: "72835eb12bf04508876fc3f19eef9dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "840e24db2484400389c16071b2471753"
I20260806 17:02:40.266644 14297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce33b2cacb143c0b5204c6b5c8e154e" candidate_uuid: "72835eb12bf04508876fc3f19eef9dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562617eb5bc74263939493c82d6467ff"
I20260806 17:02:40.266779 14297 raft_consensus.cc:3037] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:40.266773 14162 raft_consensus.cc:3037] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:40.267855 14162 raft_consensus.cc:2454] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72835eb12bf04508876fc3f19eef9dcf in term 1.
I20260806 17:02:40.267863 14297 raft_consensus.cc:2454] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72835eb12bf04508876fc3f19eef9dcf in term 1.
I20260806 17:02:40.268247 13958 leader_election.cc:302] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562617eb5bc74263939493c82d6467ff, 72835eb12bf04508876fc3f19eef9dcf; no voters:
I20260806 17:02:40.268448 14370 raft_consensus.cc:2781] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:40.268697 14370 raft_consensus.cc:686] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf [term 1 LEADER]: Becoming Leader. State: Replica: 72835eb12bf04508876fc3f19eef9dcf, State: Running, Role: LEADER
I20260806 17:02:40.269037 14370 consensus_queue.cc:237] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840e24db2484400389c16071b2471753" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 32811 } } peers { permanent_uuid: "72835eb12bf04508876fc3f19eef9dcf" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33813 } } peers { permanent_uuid: "562617eb5bc74263939493c82d6467ff" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 45689 } }
I20260806 17:02:40.272274 13879 catalog_manager.cc:5495] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 72835eb12bf04508876fc3f19eef9dcf (127.9.111.65). New cstate: current_term: 1 leader_uuid: "72835eb12bf04508876fc3f19eef9dcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840e24db2484400389c16071b2471753" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 32811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72835eb12bf04508876fc3f19eef9dcf" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562617eb5bc74263939493c82d6467ff" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 45689 } health_report { overall_health: UNKNOWN } } }
I20260806 17:02:40.287109 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.283766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40280 (local address 127.9.111.65:33813)
0806 17:02:40.283895 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:40.283899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:40.283912 (+ 13us) server_negotiation.cc:407] Connection header received
0806 17:02:40.283981 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:40.283983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:40.284033 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:40.284119 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:40.284336 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.284834 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.285653 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:40.285926 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:40.286008 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:40.286336 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:40.286802 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:40.286894 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:40.286959 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 17:02:40.287005 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 17:02:40.293006 14297 raft_consensus.cc:1261] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff [term 1 FOLLOWER]: Refusing update from remote peer 72835eb12bf04508876fc3f19eef9dcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:40.293006 14162 raft_consensus.cc:1261] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753 [term 1 FOLLOWER]: Refusing update from remote peer 72835eb12bf04508876fc3f19eef9dcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:40.293780 14373 consensus_queue.cc:1035] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "562617eb5bc74263939493c82d6467ff" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:40.294042 14370 consensus_queue.cc:1035] T fce33b2cacb143c0b5204c6b5c8e154e P 72835eb12bf04508876fc3f19eef9dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "840e24db2484400389c16071b2471753" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 32811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:40.309234 13878 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:40.309453 13878 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:40.311975 13878 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92ab87e6e42247b9be0e6906fd2fbdc9: Sending DeleteTablet for 3 replicas of tablet fce33b2cacb143c0b5204c6b5c8e154e
I20260806 17:02:40.312832 14007 tablet_service.cc:1514] Processing DeleteTablet for tablet fce33b2cacb143c0b5204c6b5c8e154e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:40 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:40270
I20260806 17:02:40.312816 14142 tablet_service.cc:1514] Processing DeleteTablet for tablet fce33b2cacb143c0b5204c6b5c8e154e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:40 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:41438
I20260806 17:02:40.313570 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 13941
I20260806 17:02:40.313519 14276 tablet_service.cc:1514] Processing DeleteTablet for tablet fce33b2cacb143c0b5204c6b5c8e154e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:40 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:39172
I20260806 17:02:40.314877 14387 tablet_replica.cc:330] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753: stopping tablet replica
I20260806 17:02:40.315197 14387 raft_consensus.cc:2229] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:40.315547 14387 raft_consensus.cc:2258] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:40.316951 14387 ts_tablet_manager.cc:1905] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:40.319131 14387 ts_tablet_manager.cc:1918] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:02:40.319211 14387 log.cc:1196] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/fce33b2cacb143c0b5204c6b5c8e154e
I20260806 17:02:40.319516 14387 ts_tablet_manager.cc:1939] T fce33b2cacb143c0b5204c6b5c8e154e P 840e24db2484400389c16071b2471753: Deleting consensus metadata
I20260806 17:02:40.320652 14388 tablet_replica.cc:330] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff: stopping tablet replica
I20260806 17:02:40.320626 13863 catalog_manager.cc:4841] TS 840e24db2484400389c16071b2471753 (127.9.111.66:32811): tablet fce33b2cacb143c0b5204c6b5c8e154e (table test-table [id=6d8efb3da32c431291e2f274d3f4d079]) successfully deleted
W20260806 17:02:40.320948 13866 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.111.65:33813 (error 108)
I20260806 17:02:40.320981 14388 raft_consensus.cc:2229] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:40.321219 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 14079
I20260806 17:02:40.321355 14388 raft_consensus.cc:2258] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:40.323410 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.323020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:33813 (local address 127.0.0.1:40292)
0806 17:02:40.323231 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:40.323320 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.65:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260806 17:02:40.323586 13866 catalog_manager.cc:4584] TS 72835eb12bf04508876fc3f19eef9dcf (127.9.111.65:33813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fce33b2cacb143c0b5204c6b5c8e154e: Network error: Client connection negotiation failed: client connection to 127.9.111.65:33813: connect: Connection refused (error 111)
I20260806 17:02:40.324592 14388 ts_tablet_manager.cc:1905] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:40.326802 14388 ts_tablet_manager.cc:1918] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:02:40.326886 14388 log.cc:1196] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/fce33b2cacb143c0b5204c6b5c8e154e
I20260806 17:02:40.327214 14388 ts_tablet_manager.cc:1939] T fce33b2cacb143c0b5204c6b5c8e154e P 562617eb5bc74263939493c82d6467ff: Deleting consensus metadata
I20260806 17:02:40.328023 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 14213
I20260806 17:02:40.334168 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 13848
2026-08-06T17:02:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2344 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T17:02:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:02:40Z Disabled control of system clock
I20260806 17:02:40.367365 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:38791
--webserver_interface=127.9.111.126
--webserver_port=0
--builtin_ntp_servers=127.9.111.84:33577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:38791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 17:02:40.479592 14395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:40.479897 14395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:40.479936 14395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:40.485229 14395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:40.485307 14395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:40.485328 14395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:40.485350 14395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:40.485366 14395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:02:40.490377 14395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:33577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:38791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:38791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:40.491453 14395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:40.492933 14395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:40.501670 14401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:40.501781 14395 server_base.cc:1034] running on GCE node
W20260806 17:02:40.501688 14400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:40.501670 14403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:40.502521 14395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:40.503759 14395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:40.504956 14395 hybrid_clock.cc:648] HybridClock initialized: now 1786035760504902 us; error 67 us; skew 500 ppm
I20260806 17:02:40.506975 14395 webserver.cc:460] Webserver started at http://127.9.111.126:38441/ using document root <none> and password file <none>
I20260806 17:02:40.507570 14395 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:40.507668 14395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:40.507900 14395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:40.510057 14395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "0108e9e1227d431192b739d6d9ab624b"
format_stamp: "Formatted at 2026-08-06 17:02:40 on dist-test-slave-69rg"
server_key: "a39323cf778bba3abd89cba8ab39a846"
server_key_iv: "04bcf3dc8c8a89e30e084a3e9c271c7a"
server_key_version: "encryptionkey@0"
I20260806 17:02:40.510623 14395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "0108e9e1227d431192b739d6d9ab624b"
format_stamp: "Formatted at 2026-08-06 17:02:40 on dist-test-slave-69rg"
server_key: "a39323cf778bba3abd89cba8ab39a846"
server_key_iv: "04bcf3dc8c8a89e30e084a3e9c271c7a"
server_key_version: "encryptionkey@0"
I20260806 17:02:40.514225 14395 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 17:02:40.516570 14410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:40.517771 14395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 17:02:40.517939 14395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "0108e9e1227d431192b739d6d9ab624b"
format_stamp: "Formatted at 2026-08-06 17:02:40 on dist-test-slave-69rg"
server_key: "a39323cf778bba3abd89cba8ab39a846"
server_key_iv: "04bcf3dc8c8a89e30e084a3e9c271c7a"
server_key_version: "encryptionkey@0"
I20260806 17:02:40.518054 14395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:40.522534 14395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:40.523115 14395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:40.523295 14395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:40.530407 14395 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:38791
I20260806 17:02:40.530423 14463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:38791 every 8 connection(s)
I20260806 17:02:40.531405 14395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:02:40.533450 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 14395
I20260806 17:02:40.533588 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:02:40.533926 9661 external_mini_cluster.cc:1428] Setting key 89b909e55da1901097a3e1828113826c
I20260806 17:02:40.535060 14464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:40.539628 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.535454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47806 (local address 127.9.111.126:38791)
0806 17:02:40.536137 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:40.536152 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:40.536188 (+ 36us) server_negotiation.cc:407] Connection header received
0806 17:02:40.536830 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:40.536846 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:40.537172 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:40.537507 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:40.538367 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:40.538372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:40.538393 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:02:40.538404 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:40.538456 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:02:40.538472 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:40.538629 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:40.538704 (+ 75us) server_negotiation.cc:299] Negotiation successful
0806 17:02:40.538955 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":151,"threads_started":1}
I20260806 17:02:40.541245 14464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b: Bootstrap starting.
I20260806 17:02:40.543572 14464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:40.544449 14464 log.cc:824] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:40.546604 14464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b: No bootstrap required, opened a new log
I20260806 17:02:40.550419 14464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0108e9e1227d431192b739d6d9ab624b" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38791 } }
I20260806 17:02:40.550650 14464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:40.550693 14464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0108e9e1227d431192b739d6d9ab624b, State: Initialized, Role: FOLLOWER
I20260806 17:02:40.551095 14464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0108e9e1227d431192b739d6d9ab624b" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38791 } }
I20260806 17:02:40.551220 14464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:02:40.551267 14464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:02:40.551347 14464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:40.552357 14464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0108e9e1227d431192b739d6d9ab624b" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38791 } }
I20260806 17:02:40.552764 14464 leader_election.cc:302] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0108e9e1227d431192b739d6d9ab624b; no voters:
I20260806 17:02:40.553144 14464 leader_election.cc:288] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:02:40.553277 14470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:40.553545 14470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b [term 1 LEADER]: Becoming Leader. State: Replica: 0108e9e1227d431192b739d6d9ab624b, State: Running, Role: LEADER
I20260806 17:02:40.553893 14470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0108e9e1227d431192b739d6d9ab624b" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38791 } }
I20260806 17:02:40.554468 14464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:40.555554 14471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0108e9e1227d431192b739d6d9ab624b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0108e9e1227d431192b739d6d9ab624b" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38791 } } }
I20260806 17:02:40.555704 14471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:40.555986 14472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0108e9e1227d431192b739d6d9ab624b. Latest consensus state: current_term: 1 leader_uuid: "0108e9e1227d431192b739d6d9ab624b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0108e9e1227d431192b739d6d9ab624b" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38791 } } }
I20260806 17:02:40.556082 14472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:40.556191 14479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:40.559293 14479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:02:40.565014 14479 catalog_manager.cc:1347] Generated new cluster ID: 1820772e48754822a1204ea3b9d9958d
I20260806 17:02:40.565102 14479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:40.601758 14479 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:02:40.602192 14479 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:40.606771 14479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b: Generated new TSK 0
I20260806 17:02:40.607600 14479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:02:40.616798 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38791
--builtin_ntp_servers=127.9.111.84:33577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 17:02:40.730687 14489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:40.731007 14489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:40.731082 14489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:40.735251 14489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:40.735349 14489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:40.735448 14489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:02:40.740389 14489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:33577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:40.741560 14489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:40.743040 14489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:40.753314 14494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:40.753332 14497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:40.753607 14489 server_base.cc:1034] running on GCE node
W20260806 17:02:40.753314 14495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:40.754238 14489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:40.755065 14489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:40.756258 14489 hybrid_clock.cc:648] HybridClock initialized: now 1786035760756228 us; error 49 us; skew 500 ppm
I20260806 17:02:40.758333 14489 webserver.cc:460] Webserver started at http://127.9.111.65:34015/ using document root <none> and password file <none>
I20260806 17:02:40.758937 14489 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:40.759032 14489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:40.759267 14489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:40.761420 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "07bc45c7db8b4b6abf1520d6e65e49f8"
format_stamp: "Formatted at 2026-08-06 17:02:40 on dist-test-slave-69rg"
server_key: "9a27e9d135a41b9113dad39ab44f4f16"
server_key_iv: "d619b3e628a4237fa2bea3d74e2f4b76"
server_key_version: "encryptionkey@0"
I20260806 17:02:40.762079 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "07bc45c7db8b4b6abf1520d6e65e49f8"
format_stamp: "Formatted at 2026-08-06 17:02:40 on dist-test-slave-69rg"
server_key: "9a27e9d135a41b9113dad39ab44f4f16"
server_key_iv: "d619b3e628a4237fa2bea3d74e2f4b76"
server_key_version: "encryptionkey@0"
I20260806 17:02:40.765978 14489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:02:40.768328 14503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:40.769483 14489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 17:02:40.769621 14489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "07bc45c7db8b4b6abf1520d6e65e49f8"
format_stamp: "Formatted at 2026-08-06 17:02:40 on dist-test-slave-69rg"
server_key: "9a27e9d135a41b9113dad39ab44f4f16"
server_key_iv: "d619b3e628a4237fa2bea3d74e2f4b76"
server_key_version: "encryptionkey@0"
I20260806 17:02:40.769742 14489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:40.774569 14489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:40.775343 14489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:40.775501 14489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:40.776032 14489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:40.776945 14489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:40.776994 14489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:40.777034 14489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:40.777051 14489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:40.786201 14489 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:33009
I20260806 17:02:40.786214 14618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:33009 every 8 connection(s)
I20260806 17:02:40.787088 14489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:02:40.792912 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.788858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:47479 (local address 127.9.111.126:38791)
0806 17:02:40.789036 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:40.789041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:40.790105 (+ 1064us) server_negotiation.cc:407] Connection header received
0806 17:02:40.791341 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:40.791343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:40.791380 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:40.791454 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:40.792545 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:40.792547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:40.792549 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:02:40.792558 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:40.792573 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:02:40.792579 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:40.792646 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:40.792766 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 17:02:40.792802 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 17:02:40.793776 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.789275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38791 (local address 127.9.111.65:47479)
0806 17:02:40.789960 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:40.790002 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:40.791156 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:40.791615 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:40.791621 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:40.791705 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:40.792315 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:02:40.792317 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:40.792356 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:02:40.792421 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:40.792651 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:40.792657 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:40.792661 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 17:02:40.792788 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 17:02:40.793018 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":110,"threads_started":1}
I20260806 17:02:40.794224 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 14489
I20260806 17:02:40.794390 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:02:40.794718 9661 external_mini_cluster.cc:1428] Setting key b00dc3fb1f8e31bb39f0f9b09e65653c
I20260806 17:02:40.795121 14619 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38791
I20260806 17:02:40.795387 14619 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:40.795799 14619 heartbeater.cc:511] Master 127.9.111.126:38791 requested a full tablet report, sending...
I20260806 17:02:40.796679 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38791
--builtin_ntp_servers=127.9.111.84:33577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 17:02:40.797345 14427 ts_manager.cc:194] Registered new tserver with Master: 07bc45c7db8b4b6abf1520d6e65e49f8 (127.9.111.65:33009)
W20260806 17:02:40.908699 14622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:40.908977 14622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:40.909024 14622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:40.913031 14622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:40.913091 14622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:40.913146 14622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:02:40.917735 14622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:33577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:40.918828 14622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:40.920141 14622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 17:02:40.930279 14622 server_base.cc:1034] running on GCE node
W20260806 17:02:40.930227 14628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:40.930227 14630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:40.930246 14627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:40.930996 14622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:40.931650 14622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:40.932863 14622 hybrid_clock.cc:648] HybridClock initialized: now 1786035760932841 us; error 35 us; skew 500 ppm
I20260806 17:02:40.934751 14622 webserver.cc:460] Webserver started at http://127.9.111.66:42801/ using document root <none> and password file <none>
I20260806 17:02:40.935360 14622 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:40.935464 14622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:40.935699 14622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:40.937872 14622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2"
format_stamp: "Formatted at 2026-08-06 17:02:40 on dist-test-slave-69rg"
server_key: "f5db74561ebbafb1889122f3c8f0dee9"
server_key_iv: "348e259e6eb9b4e71720652943c88bc3"
server_key_version: "encryptionkey@0"
I20260806 17:02:40.938478 14622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2"
format_stamp: "Formatted at 2026-08-06 17:02:40 on dist-test-slave-69rg"
server_key: "f5db74561ebbafb1889122f3c8f0dee9"
server_key_iv: "348e259e6eb9b4e71720652943c88bc3"
server_key_version: "encryptionkey@0"
I20260806 17:02:40.942221 14622 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:40.944855 14636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:40.946259 14622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:40.946393 14622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2"
format_stamp: "Formatted at 2026-08-06 17:02:40 on dist-test-slave-69rg"
server_key: "f5db74561ebbafb1889122f3c8f0dee9"
server_key_iv: "348e259e6eb9b4e71720652943c88bc3"
server_key_version: "encryptionkey@0"
I20260806 17:02:40.946507 14622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:40.951131 14622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:40.951731 14622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:40.951913 14622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:40.952471 14622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:40.953380 14622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:40.953428 14622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:40.953495 14622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:40.953534 14622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:40.962651 14622 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:36243
I20260806 17:02:40.962672 14750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:36243 every 8 connection(s)
I20260806 17:02:40.963539 14622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:02:40.969486 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.965378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:51943 (local address 127.9.111.126:38791)
0806 17:02:40.965546 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:40.965550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:40.966528 (+ 978us) server_negotiation.cc:407] Connection header received
0806 17:02:40.967800 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:40.967804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:40.967845 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:40.967921 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:40.969086 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:40.969089 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:40.969091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:02:40.969102 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:40.969121 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:02:40.969129 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:40.969206 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:40.969339 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 17:02:40.969377 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 17:02:40.970414 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:40.965683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38791 (local address 127.9.111.66:51943)
0806 17:02:40.966379 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:40.966420 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:40.967584 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:40.968083 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:40.968089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:40.968173 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:40.968832 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:02:40.968834 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:40.968872 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:02:40.968937 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:40.969202 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:40.969209 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:40.969212 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 17:02:40.969345 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 17:02:40.969532 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":161,"threads_started":1}
I20260806 17:02:40.971562 14751 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38791
I20260806 17:02:40.971791 14751 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:40.972204 14751 heartbeater.cc:511] Master 127.9.111.126:38791 requested a full tablet report, sending...
I20260806 17:02:40.973325 14427 ts_manager.cc:194] Registered new tserver with Master: 25cbdc43a93a41e9bf0a5a6d00aefca2 (127.9.111.66:36243)
I20260806 17:02:40.973802 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 14622
I20260806 17:02:40.973944 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:02:40.974272 9661 external_mini_cluster.cc:1428] Setting key dff15e7c3491859ba2bb08d9e2daf4c3
I20260806 17:02:40.976186 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38791
--builtin_ntp_servers=127.9.111.84:33577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 17:02:41.092645 14754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:41.092926 14754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:41.092967 14754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:41.097110 14754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:41.097191 14754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:41.097250 14754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:02:41.102278 14754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:33577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:41.103504 14754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:41.104913 14754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:41.115005 14759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:41.114998 14762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:41.115116 14754 server_base.cc:1034] running on GCE node
W20260806 17:02:41.115300 14760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:41.115722 14754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:41.116390 14754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:41.117620 14754 hybrid_clock.cc:648] HybridClock initialized: now 1786035761117599 us; error 32 us; skew 500 ppm
I20260806 17:02:41.119637 14754 webserver.cc:460] Webserver started at http://127.9.111.67:42959/ using document root <none> and password file <none>
I20260806 17:02:41.120265 14754 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:41.120379 14754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:41.120625 14754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:41.123070 14754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "54253c43a2a14374aebee035198d7360"
format_stamp: "Formatted at 2026-08-06 17:02:41 on dist-test-slave-69rg"
server_key: "c315483c0d3a507dc37619e87d9532a2"
server_key_iv: "c32ca4281e1f44eb031f49a26cf15e86"
server_key_version: "encryptionkey@0"
I20260806 17:02:41.123672 14754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "54253c43a2a14374aebee035198d7360"
format_stamp: "Formatted at 2026-08-06 17:02:41 on dist-test-slave-69rg"
server_key: "c315483c0d3a507dc37619e87d9532a2"
server_key_iv: "c32ca4281e1f44eb031f49a26cf15e86"
server_key_version: "encryptionkey@0"
I20260806 17:02:41.127902 14754 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 17:02:41.130689 14768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:41.132134 14754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 17:02:41.132325 14754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "54253c43a2a14374aebee035198d7360"
format_stamp: "Formatted at 2026-08-06 17:02:41 on dist-test-slave-69rg"
server_key: "c315483c0d3a507dc37619e87d9532a2"
server_key_iv: "c32ca4281e1f44eb031f49a26cf15e86"
server_key_version: "encryptionkey@0"
I20260806 17:02:41.132462 14754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:41.137717 14754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:41.138477 14754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:41.138691 14754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:41.139318 14754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:41.140349 14754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:41.140427 14754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:41.140502 14754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:41.140561 14754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:41.150297 14754 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:35859
I20260806 17:02:41.150339 14883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:35859 every 8 connection(s)
I20260806 17:02:41.151372 14754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:02:41.153719 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 14754
I20260806 17:02:41.153882 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:02:41.154186 9661 external_mini_cluster.cc:1428] Setting key e93f621627107a57e95c33c257bf1888
I20260806 17:02:41.158176 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:41.153319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:44545 (local address 127.9.111.126:38791)
0806 17:02:41.153506 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:41.153512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:41.154822 (+ 1310us) server_negotiation.cc:407] Connection header received
0806 17:02:41.156423 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:41.156426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:41.156483 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:41.156567 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:41.157719 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:41.157722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:41.157724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:02:41.157734 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:41.157751 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:02:41.157759 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:41.157885 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:41.158016 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 17:02:41.158053 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 17:02:41.159164 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:41.153768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38791 (local address 127.9.111.67:44545)
0806 17:02:41.154657 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:41.154712 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:41.156192 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:41.156705 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:41.156712 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:41.156809 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:41.157432 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:02:41.157434 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:41.157477 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:02:41.157575 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:41.157885 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:41.157892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:41.157896 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 17:02:41.158035 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 17:02:41.158281 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":153,"threads_started":1}
I20260806 17:02:41.160359 14884 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38791
I20260806 17:02:41.160602 14884 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:41.161027 14884 heartbeater.cc:511] Master 127.9.111.126:38791 requested a full tablet report, sending...
I20260806 17:02:41.162097 14427 ts_manager.cc:194] Registered new tserver with Master: 54253c43a2a14374aebee035198d7360 (127.9.111.67:35859)
I20260806 17:02:41.168545 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 17:02:41.172164 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:41.170664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47810 (local address 127.9.111.126:38791)
0806 17:02:41.170815 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:41.170818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:41.170941 (+ 123us) server_negotiation.cc:407] Connection header received
0806 17:02:41.171051 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:41.171053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:41.171085 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:41.171154 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:41.171849 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:41.171852 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:41.171854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:02:41.171865 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:41.171884 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:02:41.171891 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:41.171961 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:41.172022 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 17:02:41.172057 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 17:02:41.175489 14427 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:02:41.178825 14427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:02:41.191388 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:41.187547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:36243 (local address 127.0.0.1:45274)
0806 17:02:41.188589 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:41.188600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:41.188711 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:41.189710 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:41.189712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:41.189723 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:41.190558 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:02:41.190559 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:41.190568 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:02:41.190587 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:41.191058 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:41.191061 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:41.191064 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 17:02:41.191138 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 17:02:41.191221 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"thread_start_us":107,"threads_started":1}
I20260806 17:02:41.191390 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:41.188017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:33009 (local address 127.0.0.1:46122)
0806 17:02:41.189055 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:41.189071 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:41.189161 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:41.189674 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:41.189680 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:41.189700 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:41.190476 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:02:41.190478 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:41.190495 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:02:41.190588 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:41.191012 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:41.191022 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:41.191027 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 17:02:41.191201 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 17:02:41.191244 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"thread_start_us":102,"threads_started":1}
I20260806 17:02:41.191823 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:41.187327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:35859 (local address 127.0.0.1:33158)
0806 17:02:41.188079 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:41.188137 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:41.188242 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:41.189483 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:41.189488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:41.189563 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:41.190859 (+ 1296us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:02:41.190861 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:41.190872 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:02:41.190896 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:41.191620 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:41.191623 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:41.191626 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 17:02:41.191681 (+ 55us) client_negotiation.cc:240] Negotiation successful
0806 17:02:41.191712 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":80,"threads_started":1}
I20260806 17:02:41.191846 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:41.188171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45274 (local address 127.9.111.66:36243)
0806 17:02:41.188491 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:41.188504 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:41.188827 (+ 323us) server_negotiation.cc:407] Connection header received
0806 17:02:41.188890 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:41.188894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:41.189081 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:41.189217 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:41.190775 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:41.190779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:41.190818 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:02:41.190834 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:41.190879 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:02:41.190914 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:41.191017 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:41.191594 (+ 577us) server_negotiation.cc:299] Negotiation successful
0806 17:02:41.191671 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":139,"threads_started":1}
I20260806 17:02:41.192327 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:41.187478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33158 (local address 127.9.111.67:35859)
0806 17:02:41.187802 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:41.187818 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:41.188351 (+ 533us) server_negotiation.cc:407] Connection header received
0806 17:02:41.188415 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:41.188421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:41.188621 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:41.188756 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:41.191048 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:41.191053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:41.191063 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:02:41.191076 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:41.191121 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:02:41.191156 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:41.192029 (+ 873us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:41.192082 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 17:02:41.192148 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":130,"threads_started":1}
I20260806 17:02:41.191452 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:41.188475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46122 (local address 127.9.111.65:33009)
0806 17:02:41.188826 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:41.188844 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:41.189253 (+ 409us) server_negotiation.cc:407] Connection header received
0806 17:02:41.189360 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:41.189365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:41.189520 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:41.189653 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:41.190772 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:41.190781 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:41.190797 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:02:41.190812 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:41.190869 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:02:41.190895 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:41.191025 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:41.191219 (+ 194us) server_negotiation.cc:299] Negotiation successful
0806 17:02:41.191295 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":117,"threads_started":1}
I20260806 17:02:41.194689 14685 tablet_service.cc:1467] Processing CreateTablet for tablet b80cc4aed076451dad6ff0802f37e3f9 (DEFAULT_TABLE table=test-table [id=f6c82a2eaa014334ad07fb877af4fcf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:41.195169 14816 tablet_service.cc:1467] Processing CreateTablet for tablet b80cc4aed076451dad6ff0802f37e3f9 (DEFAULT_TABLE table=test-table [id=f6c82a2eaa014334ad07fb877af4fcf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:41.195147 14552 tablet_service.cc:1467] Processing CreateTablet for tablet b80cc4aed076451dad6ff0802f37e3f9 (DEFAULT_TABLE table=test-table [id=f6c82a2eaa014334ad07fb877af4fcf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:41.195784 14685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b80cc4aed076451dad6ff0802f37e3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:41.196095 14816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b80cc4aed076451dad6ff0802f37e3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:41.196095 14552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b80cc4aed076451dad6ff0802f37e3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:41.201956 14901 tablet_bootstrap.cc:492] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2: Bootstrap starting.
I20260806 17:02:41.203358 14902 tablet_bootstrap.cc:492] T b80cc4aed076451dad6ff0802f37e3f9 P 07bc45c7db8b4b6abf1520d6e65e49f8: Bootstrap starting.
I20260806 17:02:41.203858 14903 tablet_bootstrap.cc:492] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360: Bootstrap starting.
I20260806 17:02:41.204983 14901 tablet_bootstrap.cc:654] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:41.205534 14902 tablet_bootstrap.cc:654] T b80cc4aed076451dad6ff0802f37e3f9 P 07bc45c7db8b4b6abf1520d6e65e49f8: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:41.205855 14903 tablet_bootstrap.cc:654] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:41.205922 14901 log.cc:824] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:41.206393 14902 log.cc:824] T b80cc4aed076451dad6ff0802f37e3f9 P 07bc45c7db8b4b6abf1520d6e65e49f8: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:41.206638 14903 log.cc:824] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:41.208437 14902 tablet_bootstrap.cc:492] T b80cc4aed076451dad6ff0802f37e3f9 P 07bc45c7db8b4b6abf1520d6e65e49f8: No bootstrap required, opened a new log
I20260806 17:02:41.208541 14901 tablet_bootstrap.cc:492] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2: No bootstrap required, opened a new log
I20260806 17:02:41.208698 14902 ts_tablet_manager.cc:1397] T b80cc4aed076451dad6ff0802f37e3f9 P 07bc45c7db8b4b6abf1520d6e65e49f8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 17:02:41.208720 14903 tablet_bootstrap.cc:492] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360: No bootstrap required, opened a new log
I20260806 17:02:41.208748 14901 ts_tablet_manager.cc:1397] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 17:02:41.208879 14903 ts_tablet_manager.cc:1397] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 17:02:41.213595 14902 raft_consensus.cc:348] T b80cc4aed076451dad6ff0802f37e3f9 P 07bc45c7db8b4b6abf1520d6e65e49f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54253c43a2a14374aebee035198d7360" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35859 } } peers { permanent_uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36243 } } peers { permanent_uuid: "07bc45c7db8b4b6abf1520d6e65e49f8" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33009 } }
I20260806 17:02:41.214015 14902 raft_consensus.cc:374] T b80cc4aed076451dad6ff0802f37e3f9 P 07bc45c7db8b4b6abf1520d6e65e49f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:41.214088 14902 raft_consensus.cc:729] T b80cc4aed076451dad6ff0802f37e3f9 P 07bc45c7db8b4b6abf1520d6e65e49f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bc45c7db8b4b6abf1520d6e65e49f8, State: Initialized, Role: FOLLOWER
I20260806 17:02:41.213945 14903 raft_consensus.cc:348] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54253c43a2a14374aebee035198d7360" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35859 } } peers { permanent_uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36243 } } peers { permanent_uuid: "07bc45c7db8b4b6abf1520d6e65e49f8" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33009 } }
I20260806 17:02:41.214241 14903 raft_consensus.cc:374] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:41.214318 14903 raft_consensus.cc:729] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54253c43a2a14374aebee035198d7360, State: Initialized, Role: FOLLOWER
I20260806 17:02:41.214509 14901 raft_consensus.cc:348] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54253c43a2a14374aebee035198d7360" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35859 } } peers { permanent_uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36243 } } peers { permanent_uuid: "07bc45c7db8b4b6abf1520d6e65e49f8" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33009 } }
I20260806 17:02:41.214746 14901 raft_consensus.cc:374] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:41.214807 14901 raft_consensus.cc:729] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25cbdc43a93a41e9bf0a5a6d00aefca2, State: Initialized, Role: FOLLOWER
I20260806 17:02:41.214820 14903 consensus_queue.cc:260] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54253c43a2a14374aebee035198d7360" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35859 } } peers { permanent_uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36243 } } peers { permanent_uuid: "07bc45c7db8b4b6abf1520d6e65e49f8" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33009 } }
I20260806 17:02:41.214913 14902 consensus_queue.cc:260] T b80cc4aed076451dad6ff0802f37e3f9 P 07bc45c7db8b4b6abf1520d6e65e49f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54253c43a2a14374aebee035198d7360" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35859 } } peers { permanent_uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36243 } } peers { permanent_uuid: "07bc45c7db8b4b6abf1520d6e65e49f8" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33009 } }
I20260806 17:02:41.215231 14901 consensus_queue.cc:260] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54253c43a2a14374aebee035198d7360" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35859 } } peers { permanent_uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36243 } } peers { permanent_uuid: "07bc45c7db8b4b6abf1520d6e65e49f8" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33009 } }
I20260806 17:02:41.215656 14884 heartbeater.cc:503] Master 127.9.111.126:38791 was elected leader, sending a full tablet report...
I20260806 17:02:41.215984 14902 ts_tablet_manager.cc:1428] T b80cc4aed076451dad6ff0802f37e3f9 P 07bc45c7db8b4b6abf1520d6e65e49f8: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 17:02:41.216120 14901 ts_tablet_manager.cc:1428] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 17:02:41.216374 14619 heartbeater.cc:503] Master 127.9.111.126:38791 was elected leader, sending a full tablet report...
I20260806 17:02:41.216514 14751 heartbeater.cc:503] Master 127.9.111.126:38791 was elected leader, sending a full tablet report...
I20260806 17:02:41.217108 14903 ts_tablet_manager.cc:1428] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20260806 17:02:41.288307 14620 tablet.cc:2369] T b80cc4aed076451dad6ff0802f37e3f9 P 07bc45c7db8b4b6abf1520d6e65e49f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 17:02:41.402495 14885 tablet.cc:2369] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 17:02:41.464754 14752 tablet.cc:2369] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:02:41.634281 14907 raft_consensus.cc:482] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:41.634503 14907 raft_consensus.cc:504] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54253c43a2a14374aebee035198d7360" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35859 } } peers { permanent_uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36243 } } peers { permanent_uuid: "07bc45c7db8b4b6abf1520d6e65e49f8" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33009 } }
I20260806 17:02:41.635442 14909 raft_consensus.cc:482] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:41.635761 14909 raft_consensus.cc:504] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54253c43a2a14374aebee035198d7360" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35859 } } peers { permanent_uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36243 } } peers { permanent_uuid: "07bc45c7db8b4b6abf1520d6e65e49f8" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33009 } }
I20260806 17:02:41.635982 14907 leader_election.cc:288] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25cbdc43a93a41e9bf0a5a6d00aefca2 (127.9.111.66:36243), 07bc45c7db8b4b6abf1520d6e65e49f8 (127.9.111.65:33009)
I20260806 17:02:41.638126 14909 leader_election.cc:288] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54253c43a2a14374aebee035198d7360 (127.9.111.67:35859), 07bc45c7db8b4b6abf1520d6e65e49f8 (127.9.111.65:33009)
I20260806 17:02:41.638732 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:41.636261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:36243 (local address 127.9.111.67:47831)
0806 17:02:41.636399 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:41.636418 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:41.636511 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:41.636889 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:41.636892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:41.636909 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:41.637629 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:02:41.637631 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:41.637643 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:02:41.637668 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:41.638452 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:41.638457 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:41.638462 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 17:02:41.638553 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 17:02:41.638594 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 17:02:41.638762 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:41.636520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:47831 (local address 127.9.111.66:36243)
0806 17:02:41.636647 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:41.636652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:41.636667 (+ 15us) server_negotiation.cc:407] Connection header received
0806 17:02:41.636711 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:41.636714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:41.636757 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:41.636862 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:41.637978 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:41.637982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:41.637986 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:02:41.637997 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:41.638018 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:02:41.638029 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:41.638412 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:41.638580 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 17:02:41.638620 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 17:02:41.639431 14705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80cc4aed076451dad6ff0802f37e3f9" candidate_uuid: "54253c43a2a14374aebee035198d7360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2" is_pre_election: true
I20260806 17:02:41.639575 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:41.637565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:46605 (local address 127.9.111.67:35859)
0806 17:02:41.637674 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:41.637678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:41.637691 (+ 13us) server_negotiation.cc:407] Connection header received
0806 17:02:41.637725 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:41.637731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:41.637764 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:41.637879 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:41.639094 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:41.639098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:41.639101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:02:41.639113 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:41.639133 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:02:41.639143 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:41.639207 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:41.639388 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 17:02:41.639429 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 17:02:41.639674 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:41.636546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:38853 (local address 127.9.111.65:33009)
0806 17:02:41.636677 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:41.636682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:41.637171 (+ 489us) server_negotiation.cc:407] Connection header received
0806 17:02:41.637369 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:41.637373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:41.637417 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:41.637505 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:41.638666 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:41.638669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:41.638672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:02:41.638683 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:41.638703 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:02:41.638713 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:41.638799 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:41.639509 (+ 710us) server_negotiation.cc:299] Negotiation successful
0806 17:02:41.639553 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 17:02:41.639685 14705 raft_consensus.cc:2454] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54253c43a2a14374aebee035198d7360 in term 0.
I20260806 17:02:41.639889 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:41.637187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:35859 (local address 127.9.111.66:46605)
0806 17:02:41.637437 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:41.637499 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:41.637578 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:41.638231 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:41.638234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:41.638264 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:41.638932 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:02:41.638936 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:41.638948 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:02:41.638975 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:41.639291 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:41.639294 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:41.639296 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 17:02:41.639704 (+ 408us) client_negotiation.cc:240] Negotiation successful
0806 17:02:41.639741 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260806 17:02:41.640110 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:41.637676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:33009 (local address 127.9.111.66:60745)
0806 17:02:41.638635 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:41.638648 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:41.638716 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:41.639063 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:41.639069 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:41.639085 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:41.639518 (+ 433us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:02:41.639520 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:41.639529 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:02:41.639553 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:41.639853 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:41.639856 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:41.639859 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 17:02:41.639930 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 17:02:41.639964 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":89,"threads_started":1}
I20260806 17:02:41.640206 14772 leader_election.cc:302] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25cbdc43a93a41e9bf0a5a6d00aefca2, 54253c43a2a14374aebee035198d7360; no voters:
I20260806 17:02:41.640313 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:41.636268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:33009 (local address 127.9.111.67:38853)
0806 17:02:41.637083 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:41.637100 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:41.637231 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:41.637838 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:41.637842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:41.637857 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:41.638371 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:02:41.638374 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:41.638388 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:02:41.638419 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:41.638966 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:41.638970 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:41.638975 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 17:02:41.639056 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 17:02:41.639091 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":426,"thread_start_us":89,"threads_started":1}
I20260806 17:02:41.640542 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:41.637862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:60745 (local address 127.9.111.65:33009)
0806 17:02:41.638366 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:41.638372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:41.638806 (+ 434us) server_negotiation.cc:407] Connection header received
0806 17:02:41.638848 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:41.638851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:41.638891 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:41.638975 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:41.639682 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:41.639686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:41.639688 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:02:41.639700 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:41.639721 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:02:41.639730 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:41.639816 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:41.640378 (+ 562us) server_negotiation.cc:299] Negotiation successful
0806 17:02:41.640418 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":67,"threads_started":1}
I20260806 17:02:41.640820 14837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80cc4aed076451dad6ff0802f37e3f9" candidate_uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54253c43a2a14374aebee035198d7360" is_pre_election: true
I20260806 17:02:41.641044 14837 raft_consensus.cc:2454] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25cbdc43a93a41e9bf0a5a6d00aefca2 in term 0.
I20260806 17:02:41.641031 14572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80cc4aed076451dad6ff0802f37e3f9" candidate_uuid: "54253c43a2a14374aebee035198d7360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07bc45c7db8b4b6abf1520d6e65e49f8" is_pre_election: true
I20260806 17:02:41.641036 14571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80cc4aed076451dad6ff0802f37e3f9" candidate_uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07bc45c7db8b4b6abf1520d6e65e49f8" is_pre_election: true
I20260806 17:02:41.641330 14572 raft_consensus.cc:2454] T b80cc4aed076451dad6ff0802f37e3f9 P 07bc45c7db8b4b6abf1520d6e65e49f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54253c43a2a14374aebee035198d7360 in term 0.
I20260806 17:02:41.641319 14907 raft_consensus.cc:2781] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:02:41.641407 14907 raft_consensus.cc:482] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:02:41.641451 14907 raft_consensus.cc:3037] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:41.641482 14637 leader_election.cc:302] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25cbdc43a93a41e9bf0a5a6d00aefca2, 54253c43a2a14374aebee035198d7360; no voters:
I20260806 17:02:41.641747 14909 raft_consensus.cc:2781] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:02:41.641876 14909 raft_consensus.cc:482] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:02:41.641927 14909 raft_consensus.cc:3037] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:41.642819 14907 raft_consensus.cc:504] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54253c43a2a14374aebee035198d7360" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35859 } } peers { permanent_uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36243 } } peers { permanent_uuid: "07bc45c7db8b4b6abf1520d6e65e49f8" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33009 } }
I20260806 17:02:41.643232 14907 leader_election.cc:288] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [CANDIDATE]: Term 1 election: Requested vote from peers 25cbdc43a93a41e9bf0a5a6d00aefca2 (127.9.111.66:36243), 07bc45c7db8b4b6abf1520d6e65e49f8 (127.9.111.65:33009)
I20260806 17:02:41.643203 14909 raft_consensus.cc:504] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54253c43a2a14374aebee035198d7360" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35859 } } peers { permanent_uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36243 } } peers { permanent_uuid: "07bc45c7db8b4b6abf1520d6e65e49f8" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33009 } }
I20260806 17:02:41.643524 14705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80cc4aed076451dad6ff0802f37e3f9" candidate_uuid: "54253c43a2a14374aebee035198d7360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2"
I20260806 17:02:41.643582 14909 leader_election.cc:288] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [CANDIDATE]: Term 1 election: Requested vote from peers 54253c43a2a14374aebee035198d7360 (127.9.111.67:35859), 07bc45c7db8b4b6abf1520d6e65e49f8 (127.9.111.65:33009)
I20260806 17:02:41.643612 14571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80cc4aed076451dad6ff0802f37e3f9" candidate_uuid: "54253c43a2a14374aebee035198d7360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07bc45c7db8b4b6abf1520d6e65e49f8"
I20260806 17:02:41.643695 14705 raft_consensus.cc:2379] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 54253c43a2a14374aebee035198d7360 in current term 1: Already voted for candidate 25cbdc43a93a41e9bf0a5a6d00aefca2 in this term.
I20260806 17:02:41.643766 14571 raft_consensus.cc:3037] T b80cc4aed076451dad6ff0802f37e3f9 P 07bc45c7db8b4b6abf1520d6e65e49f8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:41.644060 14572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80cc4aed076451dad6ff0802f37e3f9" candidate_uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07bc45c7db8b4b6abf1520d6e65e49f8"
I20260806 17:02:41.644189 14837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80cc4aed076451dad6ff0802f37e3f9" candidate_uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54253c43a2a14374aebee035198d7360"
I20260806 17:02:41.644372 14837 raft_consensus.cc:2379] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 25cbdc43a93a41e9bf0a5a6d00aefca2 in current term 1: Already voted for candidate 54253c43a2a14374aebee035198d7360 in this term.
I20260806 17:02:41.644814 14637 leader_election.cc:302] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25cbdc43a93a41e9bf0a5a6d00aefca2; no voters: 07bc45c7db8b4b6abf1520d6e65e49f8, 54253c43a2a14374aebee035198d7360
I20260806 17:02:41.644975 14909 raft_consensus.cc:2726] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 17:02:41.645192 14571 raft_consensus.cc:2454] T b80cc4aed076451dad6ff0802f37e3f9 P 07bc45c7db8b4b6abf1520d6e65e49f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54253c43a2a14374aebee035198d7360 in term 1.
I20260806 17:02:41.645542 14770 leader_election.cc:302] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07bc45c7db8b4b6abf1520d6e65e49f8, 54253c43a2a14374aebee035198d7360; no voters: 25cbdc43a93a41e9bf0a5a6d00aefca2
I20260806 17:02:41.645747 14907 raft_consensus.cc:2781] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:41.646063 14907 raft_consensus.cc:686] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [term 1 LEADER]: Becoming Leader. State: Replica: 54253c43a2a14374aebee035198d7360, State: Running, Role: LEADER
I20260806 17:02:41.646392 14907 consensus_queue.cc:237] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54253c43a2a14374aebee035198d7360" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35859 } } peers { permanent_uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36243 } } peers { permanent_uuid: "07bc45c7db8b4b6abf1520d6e65e49f8" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33009 } }
I20260806 17:02:41.649952 14424 catalog_manager.cc:5495] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54253c43a2a14374aebee035198d7360 (127.9.111.67). New cstate: current_term: 1 leader_uuid: "54253c43a2a14374aebee035198d7360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54253c43a2a14374aebee035198d7360" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07bc45c7db8b4b6abf1520d6e65e49f8" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33009 } health_report { overall_health: UNKNOWN } } }
I20260806 17:02:41.759692 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:41.757676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33170 (local address 127.9.111.67:35859)
0806 17:02:41.757869 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:41.757874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:41.758053 (+ 179us) server_negotiation.cc:407] Connection header received
0806 17:02:41.758275 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:41.758278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:41.758370 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:41.758472 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:41.759256 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:41.759259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:41.759262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:02:41.759281 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:41.759306 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:02:41.759315 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:41.759412 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:41.759491 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 17:02:41.759536 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 17:02:41.765758 14571 raft_consensus.cc:1261] T b80cc4aed076451dad6ff0802f37e3f9 P 07bc45c7db8b4b6abf1520d6e65e49f8 [term 1 FOLLOWER]: Refusing update from remote peer 54253c43a2a14374aebee035198d7360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:41.765755 14705 raft_consensus.cc:1261] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [term 1 FOLLOWER]: Refusing update from remote peer 54253c43a2a14374aebee035198d7360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:41.766479 14907 consensus_queue.cc:1035] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bc45c7db8b4b6abf1520d6e65e49f8" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:41.766697 14914 consensus_queue.cc:1035] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25cbdc43a93a41e9bf0a5a6d00aefca2" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:41.776127 14920 mvcc.cc:205] Tried to move back new op lower bound from 7315602480185421824 to 7315602479706443776. Current Snapshot: MvccSnapshot[applied={T|T < 7315602480185421824}]
I20260806 17:02:41.776314 14921 mvcc.cc:205] Tried to move back new op lower bound from 7315602480185421824 to 7315602479706443776. Current Snapshot: MvccSnapshot[applied={T|T < 7315602480185421824}]
I20260806 17:02:41.783423 14427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:41.783618 14427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:41.786314 14427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0108e9e1227d431192b739d6d9ab624b: Sending DeleteTablet for 3 replicas of tablet b80cc4aed076451dad6ff0802f37e3f9
I20260806 17:02:41.787269 14685 tablet_service.cc:1514] Processing DeleteTablet for tablet b80cc4aed076451dad6ff0802f37e3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:41 UTC) from {username='slave'} at 127.0.0.1:45274
I20260806 17:02:41.787449 14816 tablet_service.cc:1514] Processing DeleteTablet for tablet b80cc4aed076451dad6ff0802f37e3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:41 UTC) from {username='slave'} at 127.0.0.1:33158
I20260806 17:02:41.787997 14929 tablet_replica.cc:330] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360: stopping tablet replica
I20260806 17:02:41.788004 14928 tablet_replica.cc:330] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2: stopping tablet replica
I20260806 17:02:41.788281 14929 raft_consensus.cc:2229] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:02:41.788305 14928 raft_consensus.cc:2229] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:41.788627 14929 raft_consensus.cc:2258] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:41.788650 14928 raft_consensus.cc:2258] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:41.790241 14928 ts_tablet_manager.cc:1905] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:41.790748 14552 tablet_service.cc:1514] Processing DeleteTablet for tablet b80cc4aed076451dad6ff0802f37e3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:41 UTC) from {username='slave'} at 127.0.0.1:46122
I20260806 17:02:41.790831 14929 ts_tablet_manager.cc:1905] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:41.791913 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 14489
I20260806 17:02:41.793322 14928 ts_tablet_manager.cc:1918] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:02:41.793423 14928 log.cc:1196] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/b80cc4aed076451dad6ff0802f37e3f9
I20260806 17:02:41.793776 14928 ts_tablet_manager.cc:1939] T b80cc4aed076451dad6ff0802f37e3f9 P 25cbdc43a93a41e9bf0a5a6d00aefca2: Deleting consensus metadata
I20260806 17:02:41.793910 14929 ts_tablet_manager.cc:1918] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:02:41.794003 14929 log.cc:1196] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/b80cc4aed076451dad6ff0802f37e3f9
I20260806 17:02:41.794322 14929 ts_tablet_manager.cc:1939] T b80cc4aed076451dad6ff0802f37e3f9 P 54253c43a2a14374aebee035198d7360: Deleting consensus metadata
I20260806 17:02:41.795135 14414 catalog_manager.cc:4841] TS 25cbdc43a93a41e9bf0a5a6d00aefca2 (127.9.111.66:36243): tablet b80cc4aed076451dad6ff0802f37e3f9 (table test-table [id=f6c82a2eaa014334ad07fb877af4fcf7]) successfully deleted
I20260806 17:02:41.796856 14411 catalog_manager.cc:4841] TS 54253c43a2a14374aebee035198d7360 (127.9.111.67:35859): tablet b80cc4aed076451dad6ff0802f37e3f9 (table test-table [id=f6c82a2eaa014334ad07fb877af4fcf7]) successfully deleted
W20260806 17:02:41.801397 14412 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.111.65:33009 (error 108)
I20260806 17:02:41.802269 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 14622
I20260806 17:02:41.803468 14932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:41.802237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:33009 (local address 127.0.0.1:46124)
0806 17:02:41.803230 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:41.803329 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.65:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":898,"thread_start_us":85,"threads_started":1}
W20260806 17:02:41.803684 14412 catalog_manager.cc:4584] TS 07bc45c7db8b4b6abf1520d6e65e49f8 (127.9.111.65:33009): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b80cc4aed076451dad6ff0802f37e3f9: Network error: Client connection negotiation failed: client connection to 127.9.111.65:33009: connect: Connection refused (error 111)
I20260806 17:02:41.808743 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 14754
I20260806 17:02:41.814708 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 14395
2026-08-06T17:02:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1479 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 17:02:41.838011 9661 mini_oidc.cc:152] Starting JWKS server
I20260806 17:02:41.839736 9661 webserver.cc:460] Webserver started at https://127.0.0.1:40923/ using document root <none> and password file <none>
I20260806 17:02:41.839936 9661 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 17:02:41.840250 9661 webserver.cc:460] Webserver started at http://127.0.0.1:33749/ using document root <none> and password file <none>
2026-08-06T17:02:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:02:41Z Disabled control of system clock
I20260806 17:02:41.851171 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:44267
--webserver_interface=127.9.111.126
--webserver_port=0
--builtin_ntp_servers=127.9.111.84:44137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:44267
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40923/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0/testchainca.cert with env {}
W20260806 17:02:41.964759 14938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:41.965029 14938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:41.965068 14938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:41.969172 14938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:41.969237 14938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:41.969254 14938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:41.969269 14938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:41.969290 14938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 17:02:41.969305 14938 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 17:02:41.969318 14938 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40923/jwks.json
I20260806 17:02:41.974686 14938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:44137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:44267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:44267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40923/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:41.975737 14938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:41.976738 14938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:41.985503 14946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:41.985553 14943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:41.985551 14944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:41.985687 14938 server_base.cc:1034] running on GCE node
I20260806 17:02:41.986421 14938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:41.987536 14938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:41.988731 14938 hybrid_clock.cc:648] HybridClock initialized: now 1786035761988713 us; error 33 us; skew 500 ppm
I20260806 17:02:41.990720 14938 webserver.cc:460] Webserver started at http://127.9.111.126:45995/ using document root <none> and password file <none>
I20260806 17:02:41.991309 14938 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:41.991407 14938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:41.991637 14938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:41.993638 14938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "7815a58cbc874cd58968b0b373225604"
format_stamp: "Formatted at 2026-08-06 17:02:41 on dist-test-slave-69rg"
server_key: "ad58f9883848b17d4d1cb5f24d298ed6"
server_key_iv: "895d13ce511ce7b7a02316c6c16dcd6d"
server_key_version: "encryptionkey@0"
I20260806 17:02:41.994341 14938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "7815a58cbc874cd58968b0b373225604"
format_stamp: "Formatted at 2026-08-06 17:02:41 on dist-test-slave-69rg"
server_key: "ad58f9883848b17d4d1cb5f24d298ed6"
server_key_iv: "895d13ce511ce7b7a02316c6c16dcd6d"
server_key_version: "encryptionkey@0"
I20260806 17:02:41.998145 14938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:02:42.000622 14952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:42.002099 14938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:42.002254 14938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "7815a58cbc874cd58968b0b373225604"
format_stamp: "Formatted at 2026-08-06 17:02:41 on dist-test-slave-69rg"
server_key: "ad58f9883848b17d4d1cb5f24d298ed6"
server_key_iv: "895d13ce511ce7b7a02316c6c16dcd6d"
server_key_version: "encryptionkey@0"
I20260806 17:02:42.002378 14938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:42.021798 14938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:42.022681 14938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:42.023098 14938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:42.031261 14938 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:44267
I20260806 17:02:42.031265 15005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:44267 every 8 connection(s)
I20260806 17:02:42.032388 14938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:02:42.035867 15006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:42.038120 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 14938
I20260806 17:02:42.038281 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:02:42.038589 9661 external_mini_cluster.cc:1428] Setting key 8772d3a212629b5767369fd86703a4fc
I20260806 17:02:42.042784 15006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604: Bootstrap starting.
I20260806 17:02:42.045130 15006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:42.046344 15006 log.cc:824] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:42.048503 15006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604: No bootstrap required, opened a new log
I20260806 17:02:42.048560 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:42.040129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48088 (local address 127.9.111.126:44267)
0806 17:02:42.040878 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:42.040894 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:42.040931 (+ 37us) server_negotiation.cc:407] Connection header received
0806 17:02:42.041773 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:42.041798 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:42.042320 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:42.042672 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:42.043154 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:42.044155 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:42.045120 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:42.045703 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:42.045928 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:42.046835 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:42.046840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:42.046854 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:02:42.047241 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:42.047306 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:02:42.047326 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:42.047436 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:42.047574 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 17:02:42.047839 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":161,"threads_started":1}
I20260806 17:02:42.051543 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:02:42.052800 15006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7815a58cbc874cd58968b0b373225604" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44267 } }
I20260806 17:02:42.053069 15006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:42.053154 15006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7815a58cbc874cd58968b0b373225604, State: Initialized, Role: FOLLOWER
I20260806 17:02:42.053599 15006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7815a58cbc874cd58968b0b373225604" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44267 } }
I20260806 17:02:42.053746 15006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:02:42.053840 15006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:02:42.053984 15006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:42.055119 15006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7815a58cbc874cd58968b0b373225604" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44267 } }
I20260806 17:02:42.055588 15006 leader_election.cc:302] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7815a58cbc874cd58968b0b373225604; no voters:
I20260806 17:02:42.055965 15006 leader_election.cc:288] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:02:42.056142 15011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:42.056424 15011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604 [term 1 LEADER]: Becoming Leader. State: Replica: 7815a58cbc874cd58968b0b373225604, State: Running, Role: LEADER
I20260806 17:02:42.056746 15011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7815a58cbc874cd58968b0b373225604" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44267 } }
I20260806 17:02:42.057369 15006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:42.058571 15013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7815a58cbc874cd58968b0b373225604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7815a58cbc874cd58968b0b373225604" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44267 } } }
I20260806 17:02:42.058729 15013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:42.058552 15014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7815a58cbc874cd58968b0b373225604. Latest consensus state: current_term: 1 leader_uuid: "7815a58cbc874cd58968b0b373225604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7815a58cbc874cd58968b0b373225604" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44267 } } }
I20260806 17:02:42.059013 15014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604 [sys.catalog]: This master's current role is: LEADER
W20260806 17:02:42.067706 15027 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:02:42.067822 15027 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:02:42.067896 15021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:42.068928 15021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:02:42.074679 15021 catalog_manager.cc:1347] Generated new cluster ID: 1b4a9a3aeebc47a9a5c5da38646b6211
I20260806 17:02:42.074766 15021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:42.081640 15021 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:02:42.082855 15021 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:42.105114 15021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7815a58cbc874cd58968b0b373225604: Generated new TSK 0
I20260806 17:02:42.106139 15021 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 17:02:42.142454 15037 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 17:02:42.142585 15037 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 17:02:42.143141 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:42.117621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48098 (local address 127.9.111.126:44267)
0806 17:02:42.117776 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:42.117782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:42.118019 (+ 237us) server_negotiation.cc:407] Connection header received
0806 17:02:42.118314 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:42.118317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:42.118372 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:42.118474 (+ 102us) server_negotiation.cc:226] Negotiated authn=JWT
0806 17:02:42.118792 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:42.119377 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:42.120641 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:42.120900 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:42.120983 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:42.121441 (+ 458us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 17:02:42.142722 (+ 21281us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 17:02:42.143008 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48098: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260806 17:02:42.143235 15009 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48098: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 17:02:42.144660 15036 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:02:42.117349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:44267 (local address 127.0.0.1:48098)
0806 17:02:42.117915 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:42.117933 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:42.118075 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:42.118522 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:42.118526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:42.118535 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0806 17:02:42.118669 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:42.118676 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:42.119498 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:42.119502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:42.120498 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:42.120509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:42.121023 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:42.121026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:42.121121 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:42.121181 (+ 60us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 17:02:42.143006 (+ 21825us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 17:02:42.143269 (+ 263us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.111.126:44267: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":136,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.9.111.126:44267: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 17:02:42.145573 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 14938
2026-08-06T17:02:42Z chronyd exiting
I20260806 17:02:42.184957 9661 test_util.cc:175] -----------------------------------------------
I20260806 17:02:42.185048 9661 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786035732873553-9661-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (348 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 17:02:42.186371 9661 mini_oidc.cc:152] Starting JWKS server
I20260806 17:02:42.187269 9661 webserver.cc:460] Webserver started at https://127.0.0.1:39101/ using document root <none> and password file <none>
I20260806 17:02:42.187362 9661 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 17:02:42.187665 9661 webserver.cc:460] Webserver started at http://127.0.0.1:42917/ using document root <none> and password file <none>
2026-08-06T17:02:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:02:42Z Disabled control of system clock
I20260806 17:02:42.197680 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:44161
--webserver_interface=127.9.111.126
--webserver_port=0
--builtin_ntp_servers=127.9.111.84:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:44161
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39101/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786035732873553-9661-0/testchainca.cert with env {}
W20260806 17:02:42.313313 15043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:42.313694 15043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:42.313774 15043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:42.318006 15043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:42.318162 15043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:42.318202 15043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:42.318241 15043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:42.318276 15043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 17:02:42.318320 15043 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 17:02:42.318352 15043 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39101/jwks.json
I20260806 17:02:42.324199 15043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:44161
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:44161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39101/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786035732873553-9661-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:42.325461 15043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:42.326553 15043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:42.335659 15051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:42.335659 15049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:42.335705 15048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:42.336078 15043 server_base.cc:1034] running on GCE node
I20260806 17:02:42.336741 15043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:42.337908 15043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:42.339151 15043 hybrid_clock.cc:648] HybridClock initialized: now 1786035762339091 us; error 47 us; skew 500 ppm
I20260806 17:02:42.341269 15043 webserver.cc:460] Webserver started at http://127.9.111.126:46181/ using document root <none> and password file <none>
I20260806 17:02:42.342007 15043 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:42.342083 15043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:42.342350 15043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:42.344584 15043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "c2b03dd6ea3e438ebbb338fcf15f16de"
format_stamp: "Formatted at 2026-08-06 17:02:42 on dist-test-slave-69rg"
server_key: "11d8af1a201ebc3a44367c064cde6967"
server_key_iv: "8882388668a1963a5a2254146d320010"
server_key_version: "encryptionkey@0"
I20260806 17:02:42.345180 15043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "c2b03dd6ea3e438ebbb338fcf15f16de"
format_stamp: "Formatted at 2026-08-06 17:02:42 on dist-test-slave-69rg"
server_key: "11d8af1a201ebc3a44367c064cde6967"
server_key_iv: "8882388668a1963a5a2254146d320010"
server_key_version: "encryptionkey@0"
I20260806 17:02:42.349030 15043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 17:02:42.351680 15057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:42.352941 15043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 17:02:42.353096 15043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "c2b03dd6ea3e438ebbb338fcf15f16de"
format_stamp: "Formatted at 2026-08-06 17:02:42 on dist-test-slave-69rg"
server_key: "11d8af1a201ebc3a44367c064cde6967"
server_key_iv: "8882388668a1963a5a2254146d320010"
server_key_version: "encryptionkey@0"
I20260806 17:02:42.353260 15043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:42.364713 15043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:42.365432 15043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:42.365639 15043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:42.372781 15043 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:44161
I20260806 17:02:42.372784 15110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:44161 every 8 connection(s)
I20260806 17:02:42.373760 15043 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:02:42.374588 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 15043
I20260806 17:02:42.374730 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:02:42.375087 9661 external_mini_cluster.cc:1428] Setting key 3bf285300a3496106e1c562c66f4434d
I20260806 17:02:42.377368 15111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:42.383486 15111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de: Bootstrap starting.
I20260806 17:02:42.384442 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:42.376435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58218 (local address 127.9.111.126:44161)
0806 17:02:42.377146 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:42.377166 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:42.377240 (+ 74us) server_negotiation.cc:407] Connection header received
0806 17:02:42.378003 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:42.378020 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:42.378426 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:42.378773 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:42.379205 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:42.380155 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:42.381119 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:42.381591 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:42.381770 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:42.382700 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:42.382707 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:42.382721 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:02:42.383087 (+ 366us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:42.383141 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:02:42.383161 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:42.383320 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:42.383479 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 17:02:42.383754 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":154,"threads_started":1}
I20260806 17:02:42.385789 15111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:42.386744 15111 log.cc:824] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:42.387359 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:02:42.388758 15111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de: No bootstrap required, opened a new log
I20260806 17:02:42.392302 15111 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b03dd6ea3e438ebbb338fcf15f16de" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44161 } }
I20260806 17:02:42.392565 15111 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:42.392635 15111 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2b03dd6ea3e438ebbb338fcf15f16de, State: Initialized, Role: FOLLOWER
I20260806 17:02:42.393132 15111 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b03dd6ea3e438ebbb338fcf15f16de" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44161 } }
I20260806 17:02:42.393301 15111 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:02:42.393390 15111 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:02:42.393514 15111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:42.394570 15111 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b03dd6ea3e438ebbb338fcf15f16de" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44161 } }
I20260806 17:02:42.394934 15111 leader_election.cc:302] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2b03dd6ea3e438ebbb338fcf15f16de; no voters:
I20260806 17:02:42.395299 15111 leader_election.cc:288] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:02:42.395455 15116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:42.395712 15116 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de [term 1 LEADER]: Becoming Leader. State: Replica: c2b03dd6ea3e438ebbb338fcf15f16de, State: Running, Role: LEADER
I20260806 17:02:42.396044 15116 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b03dd6ea3e438ebbb338fcf15f16de" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44161 } }
I20260806 17:02:42.396624 15111 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:42.397918 15117 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2b03dd6ea3e438ebbb338fcf15f16de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b03dd6ea3e438ebbb338fcf15f16de" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44161 } } }
I20260806 17:02:42.397871 15118 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2b03dd6ea3e438ebbb338fcf15f16de. Latest consensus state: current_term: 1 leader_uuid: "c2b03dd6ea3e438ebbb338fcf15f16de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b03dd6ea3e438ebbb338fcf15f16de" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44161 } } }
I20260806 17:02:42.398046 15117 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:42.398046 15118 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:42.398589 15125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:42.402156 15125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:02:42.407792 15125 catalog_manager.cc:1347] Generated new cluster ID: 321866f0fe304f2394231bda78ace683
I20260806 17:02:42.407867 15125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:42.422751 15125 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:02:42.424033 15125 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:42.434685 15125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2b03dd6ea3e438ebbb338fcf15f16de: Generated new TSK 0
I20260806 17:02:42.435570 15125 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 17:02:42.474745 15141 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 17:02:42.474851 15141 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 17:02:42.475658 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:42.452243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.9.111.126:44161)
0806 17:02:42.452405 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:42.452411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:42.452598 (+ 187us) server_negotiation.cc:407] Connection header received
0806 17:02:42.452816 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:42.452819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:42.452870 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:42.452989 (+ 119us) server_negotiation.cc:226] Negotiated authn=JWT
0806 17:02:42.453255 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:42.453798 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:42.454763 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:42.454999 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:42.455113 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:42.455473 (+ 360us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 17:02:42.475125 (+ 19652us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39101/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 17:02:42.475480 (+ 355us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58228: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39101/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 17:02:42.475752 15114 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58228: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39101/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 17:02:42.475818 15140 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:02:42.452114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:44161 (local address 127.0.0.1:58228)
0806 17:02:42.452503 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:42.452516 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:42.452652 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:42.452990 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:42.452993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:42.453002 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0806 17:02:42.453126 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:42.453133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:42.453965 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:42.453969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:42.454627 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:42.454636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:42.455137 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:42.455141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:42.455242 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:42.455268 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 17:02:42.475368 (+ 20100us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39101/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 17:02:42.475586 (+ 218us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.111.126:44161: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39101/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":127,"threads_started":1}
I20260806 17:02:42.476737 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 15043
2026-08-06T17:02:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (331 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 17:02:42.517534 9661 mini_oidc.cc:152] Starting JWKS server
I20260806 17:02:42.518554 9661 webserver.cc:460] Webserver started at https://127.0.0.1:33439/ using document root <none> and password file <none>
I20260806 17:02:42.518702 9661 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 17:02:42.518986 9661 webserver.cc:460] Webserver started at http://127.0.0.1:44849/ using document root <none> and password file <none>
2026-08-06T17:02:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:02:42Z Disabled control of system clock
I20260806 17:02:42.529887 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:45299
--webserver_interface=127.9.111.126
--webserver_port=0
--builtin_ntp_servers=127.9.111.84:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:45299
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33439/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 17:02:42.644289 15147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:42.644591 15147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:42.644632 15147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:42.648862 15147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:42.648948 15147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:42.648965 15147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:42.648984 15147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:42.648999 15147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 17:02:42.649016 15147 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 17:02:42.649080 15147 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33439/jwks.json
I20260806 17:02:42.654798 15147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:45299
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:45299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33439/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:42.656042 15147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:42.657534 15147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:42.666181 15153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:42.666186 15156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:42.666186 15154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:42.666507 15147 server_base.cc:1034] running on GCE node
I20260806 17:02:42.667292 15147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:42.668426 15147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:42.669622 15147 hybrid_clock.cc:648] HybridClock initialized: now 1786035762669570 us; error 50 us; skew 500 ppm
I20260806 17:02:42.671700 15147 webserver.cc:460] Webserver started at http://127.9.111.126:37105/ using document root <none> and password file <none>
I20260806 17:02:42.672299 15147 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:42.672367 15147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:42.672588 15147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:42.674830 15147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "d0f501ae777343a3b2a6184915e0fc67"
format_stamp: "Formatted at 2026-08-06 17:02:42 on dist-test-slave-69rg"
server_key: "032476f57313844a6394a0dffb805285"
server_key_iv: "7c9ca3bf5224e774a6518b47c8df72de"
server_key_version: "encryptionkey@0"
I20260806 17:02:42.675428 15147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "d0f501ae777343a3b2a6184915e0fc67"
format_stamp: "Formatted at 2026-08-06 17:02:42 on dist-test-slave-69rg"
server_key: "032476f57313844a6394a0dffb805285"
server_key_iv: "7c9ca3bf5224e774a6518b47c8df72de"
server_key_version: "encryptionkey@0"
I20260806 17:02:42.679476 15147 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:02:42.682125 15162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:42.683424 15147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 17:02:42.683564 15147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "d0f501ae777343a3b2a6184915e0fc67"
format_stamp: "Formatted at 2026-08-06 17:02:42 on dist-test-slave-69rg"
server_key: "032476f57313844a6394a0dffb805285"
server_key_iv: "7c9ca3bf5224e774a6518b47c8df72de"
server_key_version: "encryptionkey@0"
I20260806 17:02:42.683687 15147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:42.714437 15147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:42.715215 15147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:42.715445 15147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:42.723129 15147 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:45299
I20260806 17:02:42.723150 15216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:45299 every 8 connection(s)
I20260806 17:02:42.724170 15147 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:02:42.726922 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 15147
I20260806 17:02:42.727061 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:02:42.727305 15217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:42.727391 9661 external_mini_cluster.cc:1428] Setting key 290e5cdf5939ae6049be8af5d1aa78af
I20260806 17:02:42.735180 15217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67: Bootstrap starting.
I20260806 17:02:42.738188 15217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:42.738325 15220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:42.728823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41724 (local address 127.9.111.126:45299)
0806 17:02:42.729526 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:42.729549 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:42.729593 (+ 44us) server_negotiation.cc:407] Connection header received
0806 17:02:42.730292 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:42.730323 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:42.730838 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:42.731217 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:42.731684 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:42.733148 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:42.734250 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:42.734945 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:42.735114 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:42.736098 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:42.736104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:42.736119 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:02:42.736647 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:42.736720 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:02:42.736747 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:42.736847 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:42.737036 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 17:02:42.737317 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":126,"threads_started":1}
I20260806 17:02:42.739068 15217 log.cc:824] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:42.741281 15217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67: No bootstrap required, opened a new log
I20260806 17:02:42.741417 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:02:42.745167 15217 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f501ae777343a3b2a6184915e0fc67" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45299 } }
I20260806 17:02:42.745414 15217 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:42.745450 15217 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f501ae777343a3b2a6184915e0fc67, State: Initialized, Role: FOLLOWER
I20260806 17:02:42.745955 15217 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f501ae777343a3b2a6184915e0fc67" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45299 } }
I20260806 17:02:42.746097 15217 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:02:42.746182 15217 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:02:42.746304 15217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:42.747282 15217 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f501ae777343a3b2a6184915e0fc67" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45299 } }
I20260806 17:02:42.747583 15217 leader_election.cc:302] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0f501ae777343a3b2a6184915e0fc67; no voters:
I20260806 17:02:42.747869 15217 leader_election.cc:288] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:02:42.748015 15222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:42.748309 15222 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67 [term 1 LEADER]: Becoming Leader. State: Replica: d0f501ae777343a3b2a6184915e0fc67, State: Running, Role: LEADER
I20260806 17:02:42.748631 15222 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f501ae777343a3b2a6184915e0fc67" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45299 } }
I20260806 17:02:42.749207 15217 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:42.750548 15223 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0f501ae777343a3b2a6184915e0fc67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f501ae777343a3b2a6184915e0fc67" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45299 } } }
I20260806 17:02:42.750548 15224 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0f501ae777343a3b2a6184915e0fc67. Latest consensus state: current_term: 1 leader_uuid: "d0f501ae777343a3b2a6184915e0fc67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f501ae777343a3b2a6184915e0fc67" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45299 } } }
I20260806 17:02:42.750720 15224 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:42.750720 15223 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:42.751217 15233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:42.754496 15233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:02:42.759938 15233 catalog_manager.cc:1347] Generated new cluster ID: d1573889d59a4bcdad0b79301ec5e714
I20260806 17:02:42.760028 15233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:42.772089 15233 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:02:42.773092 15233 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:42.786669 15233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0f501ae777343a3b2a6184915e0fc67: Generated new TSK 0
I20260806 17:02:42.787578 15233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:02:42.808964 15220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:42.803323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41728 (local address 127.9.111.126:45299)
0806 17:02:42.803476 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:42.803481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:42.803633 (+ 152us) server_negotiation.cc:407] Connection header received
0806 17:02:42.803802 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:42.803804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:42.803863 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:42.803988 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0806 17:02:42.804253 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:42.805103 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:42.806155 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:42.806529 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:42.806639 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:42.807014 (+ 375us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 17:02:42.808542 (+ 1528us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33439/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 17:02:42.808797 (+ 255us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41728: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33439/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 17:02:42.809051 15220 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41728: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33439/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 17:02:42.810628 15247 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:02:42.803186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:45299 (local address 127.0.0.1:41728)
0806 17:02:42.803543 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:42.803556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:42.803659 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:42.804020 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:42.804023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:42.804030 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0806 17:02:42.804134 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:42.804139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:42.805252 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:42.805255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:42.806027 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:42.806037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:42.806699 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:42.806702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:42.806789 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:42.806815 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 17:02:42.808722 (+ 1907us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33439/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 17:02:42.810447 (+ 1725us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.111.126:45299: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33439/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":1484,"thread_start_us":117,"threads_started":1}
I20260806 17:02:42.811203 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 15147
E20260806 17:02:42.819918 15248 webserver.cc:533] Webserver: sslize failed:
E20260806 17:02:42.820032 15248 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-06T17:02:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (334 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T17:02:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:02:42Z Disabled control of system clock
I20260806 17:02:42.862198 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:36337
--webserver_interface=127.9.111.126
--webserver_port=0
--builtin_ntp_servers=127.9.111.84:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:36337
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786035732873553-9661-0/testchainca.cert with env {}
W20260806 17:02:42.975765 15252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:42.976217 15252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:42.976295 15252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:42.983407 15252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:42.983516 15252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:42.983542 15252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:42.983593 15252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:42.983626 15252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 17:02:42.983660 15252 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 17:02:42.983680 15252 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 17:02:42.993381 15252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:36337
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:36337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786035732873553-9661-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:42.995071 15252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:42.996564 15252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:43.008505 15261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:43.008482 15259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:43.008729 15252 server_base.cc:1034] running on GCE node
W20260806 17:02:43.008737 15258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:43.009477 15252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:43.011210 15252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:43.012436 15252 hybrid_clock.cc:648] HybridClock initialized: now 1786035763012421 us; error 37 us; skew 500 ppm
I20260806 17:02:43.015357 15252 webserver.cc:460] Webserver started at http://127.9.111.126:34255/ using document root <none> and password file <none>
I20260806 17:02:43.016252 15252 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:43.016372 15252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:43.016662 15252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:43.020067 15252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "b2ffde2ea2604e3c8a0fc50dc374ad27"
format_stamp: "Formatted at 2026-08-06 17:02:43 on dist-test-slave-69rg"
server_key: "84a147684797abdc6504f8e58ae0fcc0"
server_key_iv: "4ca77c14f8590c24338a0a4ca486b859"
server_key_version: "encryptionkey@0"
I20260806 17:02:43.020903 15252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "b2ffde2ea2604e3c8a0fc50dc374ad27"
format_stamp: "Formatted at 2026-08-06 17:02:43 on dist-test-slave-69rg"
server_key: "84a147684797abdc6504f8e58ae0fcc0"
server_key_iv: "4ca77c14f8590c24338a0a4ca486b859"
server_key_version: "encryptionkey@0"
I20260806 17:02:43.026285 15252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 17:02:43.029305 15267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:43.030782 15252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:43.030936 15252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "b2ffde2ea2604e3c8a0fc50dc374ad27"
format_stamp: "Formatted at 2026-08-06 17:02:43 on dist-test-slave-69rg"
server_key: "84a147684797abdc6504f8e58ae0fcc0"
server_key_iv: "4ca77c14f8590c24338a0a4ca486b859"
server_key_version: "encryptionkey@0"
I20260806 17:02:43.031059 15252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:43.070149 15252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:43.070955 15252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:43.071188 15252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:43.078893 15252 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:36337
I20260806 17:02:43.078912 15323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:36337 every 8 connection(s)
I20260806 17:02:43.079949 15252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:02:43.083621 15324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:43.089509 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 15252
I20260806 17:02:43.089495 15324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27: Bootstrap starting.
I20260806 17:02:43.089644 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:02:43.089993 9661 external_mini_cluster.cc:1428] Setting key ae8b6d426dbd81f64f2ed2cfa0cad6ea
I20260806 17:02:43.092368 15324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:43.093178 15324 log.cc:824] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:43.095436 15324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27: No bootstrap required, opened a new log
I20260806 17:02:43.099956 15324 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ffde2ea2604e3c8a0fc50dc374ad27" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36337 } }
I20260806 17:02:43.100229 15324 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:43.100314 15324 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2ffde2ea2604e3c8a0fc50dc374ad27, State: Initialized, Role: FOLLOWER
I20260806 17:02:43.100539 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:43.091515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.9.111.126:36337)
0806 17:02:43.092236 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:43.092251 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:43.092289 (+ 38us) server_negotiation.cc:407] Connection header received
0806 17:02:43.093084 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:43.093102 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:43.093624 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:43.094054 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:43.094530 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:43.095675 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:43.096885 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:43.097391 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:43.097517 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:43.098484 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:43.098490 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:43.098503 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:02:43.098930 (+ 427us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:43.098991 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:02:43.099012 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:43.099097 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:43.099282 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 17:02:43.099611 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":151,"threads_started":1}
I20260806 17:02:43.100842 15324 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ffde2ea2604e3c8a0fc50dc374ad27" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36337 } }
I20260806 17:02:43.100978 15324 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:02:43.101065 15324 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:02:43.101190 15324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:43.102488 15324 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ffde2ea2604e3c8a0fc50dc374ad27" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36337 } }
I20260806 17:02:43.102910 15324 leader_election.cc:302] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2ffde2ea2604e3c8a0fc50dc374ad27; no voters:
I20260806 17:02:43.103304 15324 leader_election.cc:288] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:02:43.103389 15330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:43.103586 15330 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27 [term 1 LEADER]: Becoming Leader. State: Replica: b2ffde2ea2604e3c8a0fc50dc374ad27, State: Running, Role: LEADER
I20260806 17:02:43.104033 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:02:43.104679 15324 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:43.103900 15330 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ffde2ea2604e3c8a0fc50dc374ad27" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36337 } }
I20260806 17:02:43.106922 15332 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2ffde2ea2604e3c8a0fc50dc374ad27. Latest consensus state: current_term: 1 leader_uuid: "b2ffde2ea2604e3c8a0fc50dc374ad27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ffde2ea2604e3c8a0fc50dc374ad27" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36337 } } }
I20260806 17:02:43.106980 15331 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2ffde2ea2604e3c8a0fc50dc374ad27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ffde2ea2604e3c8a0fc50dc374ad27" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36337 } } }
I20260806 17:02:43.107352 15332 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:43.107353 15331 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27 [sys.catalog]: This master's current role is: LEADER
W20260806 17:02:43.109591 15344 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:02:43.109661 15344 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:02:43.109748 15345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:43.110491 15345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:02:43.117077 15345 catalog_manager.cc:1347] Generated new cluster ID: 13c3c4e27a8d4ab497b676a6c85e6723
I20260806 17:02:43.117158 15345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:43.130733 15345 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:02:43.131721 15345 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:43.141369 15345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2ffde2ea2604e3c8a0fc50dc374ad27: Generated new TSK 0
I20260806 17:02:43.142256 15345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:02:43.668470 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:43.158164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51014 (local address 127.9.111.126:36337)
0806 17:02:43.158343 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:43.158348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:43.158625 (+ 277us) server_negotiation.cc:407] Connection header received
0806 17:02:43.158802 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:43.158805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:43.158865 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:43.158968 (+ 103us) server_negotiation.cc:226] Negotiated authn=JWT
0806 17:02:43.159255 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:43.159855 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:43.161133 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:43.161361 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:43.161447 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:43.161810 (+ 363us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 17:02:43.668038 (+506228us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 17:02:43.668339 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51014: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":63}
W20260806 17:02:43.668596 15327 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51014: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 17:02:43.668721 15355 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:02:43.158068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:36337 (local address 127.0.0.1:51014)
0806 17:02:43.158497 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:43.158513 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:43.158652 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:43.158996 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:43.158999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:43.159009 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0806 17:02:43.159138 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:43.159146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:43.159974 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:43.159980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:43.160993 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:43.161006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:43.161490 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:43.161494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:43.161591 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:43.161621 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 17:02:43.668297 (+506676us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 17:02:43.668518 (+ 221us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.111.126:36337: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":158,"threads_started":1}
I20260806 17:02:43.669428 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 15252
2026-08-06T17:02:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (837 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (128 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (126 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:02:43 dist-test-slave-69rg krb5kdc[15367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:02:43 dist-test-slave-69rg krb5kdc[15367](info): set up 2 sockets
Aug 06 17:02:43 dist-test-slave-69rg krb5kdc[15367](info): commencing operation
krb5kdc: starting...
W20260806 17:02:45.073258 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:02:45 dist-test-slave-69rg krb5kdc[15367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:02:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:02:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:02:45.208683 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:38431
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:44995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:38431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:45.322502 15380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:45.322846 15380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:45.322932 15380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:45.327030 15380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:45.327162 15380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:45.327199 15380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:45.327243 15380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:45.327281 15380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:02:45.332921 15380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:44995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:38431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:38431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:45.334255 15380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:45.335431 15380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:45.344378 15388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:45.344594 15380 server_base.cc:1034] running on GCE node
W20260806 17:02:45.344372 15386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:45.344381 15385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:45.345304 15380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:45.346557 15380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:45.347779 15380 hybrid_clock.cc:648] HybridClock initialized: now 1786035765347762 us; error 35 us; skew 500 ppm
Aug 06 17:02:45 dist-test-slave-69rg krb5kdc[15367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035765, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:45.356175 15380 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:02:45.357275 15380 webserver.cc:460] Webserver started at http://127.9.111.126:37571/ using document root <none> and password file <none>
I20260806 17:02:45.357990 15380 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:45.358109 15380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:45.358383 15380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:45.360709 15380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "7f24458949c34e71960145ccb12058fd"
format_stamp: "Formatted at 2026-08-06 17:02:45 on dist-test-slave-69rg"
server_key: "417f5cadee2d37ef93c6536437584f74"
server_key_iv: "895dbb0e052d2d6646636005d4d401c7"
server_key_version: "encryptionkey@0"
I20260806 17:02:45.361366 15380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "7f24458949c34e71960145ccb12058fd"
format_stamp: "Formatted at 2026-08-06 17:02:45 on dist-test-slave-69rg"
server_key: "417f5cadee2d37ef93c6536437584f74"
server_key_iv: "895dbb0e052d2d6646636005d4d401c7"
server_key_version: "encryptionkey@0"
I20260806 17:02:45.365520 15380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 17:02:45.368150 15395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:45.369493 15380 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 17:02:45.369652 15380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "7f24458949c34e71960145ccb12058fd"
format_stamp: "Formatted at 2026-08-06 17:02:45 on dist-test-slave-69rg"
server_key: "417f5cadee2d37ef93c6536437584f74"
server_key_iv: "895dbb0e052d2d6646636005d4d401c7"
server_key_version: "encryptionkey@0"
I20260806 17:02:45.369778 15380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:45.397650 15380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:45.401268 15380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:45.401520 15380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:45.409464 15380 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:38431
I20260806 17:02:45.409492 15449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:38431 every 8 connection(s)
I20260806 17:02:45.410656 15380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:02:45.413524 15450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:45.416625 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 15380
I20260806 17:02:45.416792 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:02:45.417094 9661 external_mini_cluster.cc:1428] Setting key 6b557687c4071dc5b9ec794e1d72655e
I20260806 17:02:45.420009 15450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd: Bootstrap starting.
I20260806 17:02:45.422338 15450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:45.423190 15450 log.cc:824] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:45.425756 15450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd: No bootstrap required, opened a new log
Aug 06 17:02:45 dist-test-slave-69rg krb5kdc[15367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:45.430642 15450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f24458949c34e71960145ccb12058fd" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38431 } }
I20260806 17:02:45.430968 15450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:45.431016 15450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f24458949c34e71960145ccb12058fd, State: Initialized, Role: FOLLOWER
I20260806 17:02:45.431542 15450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f24458949c34e71960145ccb12058fd" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38431 } }
I20260806 17:02:45.431701 15450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:02:45.431774 15450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:02:45.431880 15450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:45.433120 15450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f24458949c34e71960145ccb12058fd" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38431 } }
I20260806 17:02:45.433491 15450 leader_election.cc:302] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f24458949c34e71960145ccb12058fd; no voters:
I20260806 17:02:45.433746 15453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:45.418686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48004 (local address 127.9.111.126:38431)
0806 17:02:45.419182 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:45.419198 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:45.419235 (+ 37us) server_negotiation.cc:407] Connection header received
0806 17:02:45.419920 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:45.419938 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:45.420344 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:45.420703 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:45.421615 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:45.422883 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:45.424112 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:45.424832 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:45.425009 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:45.427856 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:45.427863 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:45.427882 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:45.427916 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:45.430154 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:45.430675 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:45.430682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:45.430688 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:45.430831 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:45.431262 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:45.431266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:45.431268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:45.431825 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:45.432064 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:45.432324 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 17:02:45.432632 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":160,"threads_started":1}
I20260806 17:02:45.433876 15450 leader_election.cc:288] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:02:45.434062 15456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:45.434343 15456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd [term 1 LEADER]: Becoming Leader. State: Replica: 7f24458949c34e71960145ccb12058fd, State: Running, Role: LEADER
I20260806 17:02:45.434752 15456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f24458949c34e71960145ccb12058fd" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38431 } }
I20260806 17:02:45.435181 15450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:45.436470 15458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f24458949c34e71960145ccb12058fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f24458949c34e71960145ccb12058fd" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38431 } } }
I20260806 17:02:45.436589 15458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:45.436828 15457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f24458949c34e71960145ccb12058fd. Latest consensus state: current_term: 1 leader_uuid: "7f24458949c34e71960145ccb12058fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f24458949c34e71960145ccb12058fd" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38431 } } }
I20260806 17:02:45.436936 15457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:45.437150 15465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:45.440861 15465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:02:45.447371 15465 catalog_manager.cc:1347] Generated new cluster ID: ab65171d7dd84dbdb2de3d0cb6089a3e
I20260806 17:02:45.447461 15465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:45.456372 15465 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:02:45.457391 15465 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:45.465557 15465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f24458949c34e71960145ccb12058fd: Generated new TSK 0
I20260806 17:02:45.466334 15465 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
I20260806 17:02:45.498931 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38431
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:44995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:45.616971 15477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:45.617306 15477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:45.617401 15477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:45.621578 15477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:45.621676 15477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:45.621767 15477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:02:45.627163 15477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:44995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:45.628314 15477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:45.629359 15477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:45.639362 15482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:45.639351 15483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:45.639487 15477 server_base.cc:1034] running on GCE node
W20260806 17:02:45.639356 15486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:45.640064 15477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:45.640717 15477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:45.641896 15477 hybrid_clock.cc:648] HybridClock initialized: now 1786035765641887 us; error 27 us; skew 500 ppm
Aug 06 17:02:45 dist-test-slave-69rg krb5kdc[15367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035765, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:45.644768 15477 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:02:45.645644 15477 webserver.cc:460] Webserver started at http://127.9.111.65:38689/ using document root <none> and password file <none>
I20260806 17:02:45.646248 15477 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:45.646323 15477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:45.646513 15477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:45.648545 15477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "ea3c6891d3c14ad894aa9fee31a05133"
format_stamp: "Formatted at 2026-08-06 17:02:45 on dist-test-slave-69rg"
server_key: "b5b660c0d4d90f8d580ae887e4d76a8e"
server_key_iv: "db4552104e3969e92a659bd89db5d58a"
server_key_version: "encryptionkey@0"
I20260806 17:02:45.649142 15477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "ea3c6891d3c14ad894aa9fee31a05133"
format_stamp: "Formatted at 2026-08-06 17:02:45 on dist-test-slave-69rg"
server_key: "b5b660c0d4d90f8d580ae887e4d76a8e"
server_key_iv: "db4552104e3969e92a659bd89db5d58a"
server_key_version: "encryptionkey@0"
I20260806 17:02:45.652981 15477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 17:02:45.655753 15493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:45.657094 15477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 17:02:45.657203 15477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "ea3c6891d3c14ad894aa9fee31a05133"
format_stamp: "Formatted at 2026-08-06 17:02:45 on dist-test-slave-69rg"
server_key: "b5b660c0d4d90f8d580ae887e4d76a8e"
server_key_iv: "db4552104e3969e92a659bd89db5d58a"
server_key_version: "encryptionkey@0"
I20260806 17:02:45.657301 15477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:45.678858 15477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:45.682190 15477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:45.682392 15477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:45.683055 15477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:45.683981 15477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:45.684031 15477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:45.684072 15477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:45.684088 15477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:45.693332 15477 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:37621
I20260806 17:02:45.693357 15608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:37621 every 8 connection(s)
I20260806 17:02:45.694300 15477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:02:45.697379 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 15477
I20260806 17:02:45.697518 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:02:45.697867 9661 external_mini_cluster.cc:1428] Setting key 9f9c4aeafef325a77220c2adcefd40a4
Aug 06 17:02:45 dist-test-slave-69rg krb5kdc[15367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035765, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.111.66@KRBTEST.COM; defaulting to no policy
I20260806 17:02:45.708557 15453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:45.696089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:50633 (local address 127.9.111.126:38431)
0806 17:02:45.696249 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:45.696261 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:45.697080 (+ 819us) server_negotiation.cc:407] Connection header received
0806 17:02:45.698744 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:45.698747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:45.698804 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:45.698882 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:45.700054 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:45.700595 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:45.701759 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:45.702064 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:45.702178 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:45.704892 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:45.704895 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:45.704897 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:45.704922 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:45.706467 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:45.706992 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:45.706997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:45.706999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:45.707081 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:45.707419 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:45.707424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:45.707425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:45.707611 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:45.707729 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:45.708235 (+ 506us) server_negotiation.cc:299] Negotiation successful
0806 17:02:45.708373 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":60}
I20260806 17:02:45.709460 15611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:45.696475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38431 (local address 127.9.111.65:50633)
0806 17:02:45.696931 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:45.696973 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:45.698512 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:45.699059 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:45.699066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:45.699430 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:45.699904 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:45.699917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:45.700737 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:45.700740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:45.701622 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:45.701631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:45.702219 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:45.702223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:45.702397 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:45.703035 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:45.703052 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:45.704713 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:45.706598 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:45.706603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:45.706618 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:45.706884 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:45.707182 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:45.707185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:45.707187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:45.707323 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:45.707727 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:45.707734 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:45.707991 (+ 257us) client_negotiation.cc:769] Sending connection context
0806 17:02:45.708192 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 17:02:45.708529 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.66@KRBTEST.COM" created.
I20260806 17:02:45.711035 15609 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38431
I20260806 17:02:45.711297 15609 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:45.711833 15609 heartbeater.cc:511] Master 127.9.111.126:38431 requested a full tablet report, sending...
I20260806 17:02:45.713625 15412 ts_manager.cc:194] Registered new tserver with Master: ea3c6891d3c14ad894aa9fee31a05133 (127.9.111.65:37621)
I20260806 17:02:45.715344 15412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:50633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
I20260806 17:02:45.728983 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38431
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:44995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:45.842465 15615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:45.842744 15615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:45.842792 15615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:45.847210 15615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:45.847292 15615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:45.847362 15615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:02:45.853137 15615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:44995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:45.854593 15615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:45.855762 15615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:45.866618 15621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:45.866850 15624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:45.866608 15622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:45.867409 15615 server_base.cc:1034] running on GCE node
I20260806 17:02:45.867895 15615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:45.868607 15615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:45.869803 15615 hybrid_clock.cc:648] HybridClock initialized: now 1786035765869788 us; error 30 us; skew 500 ppm
Aug 06 17:02:45 dist-test-slave-69rg krb5kdc[15367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035765, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:45.872941 15615 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:02:45.873951 15615 webserver.cc:460] Webserver started at http://127.9.111.66:34607/ using document root <none> and password file <none>
I20260806 17:02:45.874540 15615 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:45.874635 15615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:45.874883 15615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:45.877074 15615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "506c2814896242699a4246d8501fddd5"
format_stamp: "Formatted at 2026-08-06 17:02:45 on dist-test-slave-69rg"
server_key: "6bbba15ea1b34a87eb4152ae4f3955bd"
server_key_iv: "07bebf9fb8b10b4c41f920129a59d3e8"
server_key_version: "encryptionkey@0"
I20260806 17:02:45.877665 15615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "506c2814896242699a4246d8501fddd5"
format_stamp: "Formatted at 2026-08-06 17:02:45 on dist-test-slave-69rg"
server_key: "6bbba15ea1b34a87eb4152ae4f3955bd"
server_key_iv: "07bebf9fb8b10b4c41f920129a59d3e8"
server_key_version: "encryptionkey@0"
I20260806 17:02:45.881665 15615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 17:02:45.884364 15631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:45.885612 15615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260806 17:02:45.885768 15615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "506c2814896242699a4246d8501fddd5"
format_stamp: "Formatted at 2026-08-06 17:02:45 on dist-test-slave-69rg"
server_key: "6bbba15ea1b34a87eb4152ae4f3955bd"
server_key_iv: "07bebf9fb8b10b4c41f920129a59d3e8"
server_key_version: "encryptionkey@0"
I20260806 17:02:45.886034 15615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:45.912510 15615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:45.916131 15615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:45.916365 15615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:45.917011 15615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:45.917982 15615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:45.918033 15615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:45.918076 15615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:45.918154 15615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:45.927577 15615 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:37485
I20260806 17:02:45.927594 15744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:37485 every 8 connection(s)
I20260806 17:02:45.928557 15615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:02:45.937134 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 15615
I20260806 17:02:45.937252 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:02:45.937610 9661 external_mini_cluster.cc:1428] Setting key 41918b748b9960adc16b788465137f97
Aug 06 17:02:45 dist-test-slave-69rg krb5kdc[15367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035765, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:45.945209 15453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:45.930589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:38865 (local address 127.9.111.126:38431)
0806 17:02:45.930781 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:45.930785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:45.931733 (+ 948us) server_negotiation.cc:407] Connection header received
0806 17:02:45.933141 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:45.933144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:45.933195 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:45.933277 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:45.934654 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:45.935206 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:45.936607 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:45.936844 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:45.936983 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:45.940581 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:45.940591 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:45.940594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:45.940624 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:45.942519 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:45.943118 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:45.943124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:45.943129 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:45.943230 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:45.943580 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:45.943586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:45.943590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:45.943886 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:45.943996 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:45.944614 (+ 618us) server_negotiation.cc:299] Negotiation successful
0806 17:02:45.944786 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 17:02:45.945993 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:45.930874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38431 (local address 127.9.111.66:38865)
0806 17:02:45.931559 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:45.931607 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:45.932899 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:45.933479 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:45.933486 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:45.933920 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:45.934464 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:45.934479 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:45.935428 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:45.935432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:45.936436 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:45.936449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:45.937032 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:45.937036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:45.937248 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:45.938341 (+ 1093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:45.938370 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:45.940378 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:45.942665 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:45.942671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:45.942686 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:45.943003 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:45.943332 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:45.943334 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:45.943335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:45.943482 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:45.944014 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:45.944020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:45.944322 (+ 302us) client_negotiation.cc:769] Sending connection context
0806 17:02:45.944513 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 17:02:45.944899 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":127,"threads_started":1}
I20260806 17:02:45.947528 15745 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38431
I20260806 17:02:45.947804 15745 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:45.948342 15745 heartbeater.cc:511] Master 127.9.111.126:38431 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.111.67@KRBTEST.COM; defaulting to no policy
I20260806 17:02:45.949779 15412 ts_manager.cc:194] Registered new tserver with Master: 506c2814896242699a4246d8501fddd5 (127.9.111.66:37485)
I20260806 17:02:45.950420 15412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:38865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
I20260806 17:02:45.974799 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38431
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:44995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:46.097718 15750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:46.098129 15750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:46.098220 15750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:46.103168 15750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:46.103286 15750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:46.103391 15750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:02:46.109208 15750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:44995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:46.110514 15750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:46.111604 15750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:46.122469 15757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:46.122622 15759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:46.122460 15756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:46.122754 15750 server_base.cc:1034] running on GCE node
I20260806 17:02:46.123288 15750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:46.123992 15750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:46.125190 15750 hybrid_clock.cc:648] HybridClock initialized: now 1786035766125158 us; error 41 us; skew 500 ppm
Aug 06 17:02:46 dist-test-slave-69rg krb5kdc[15367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035766, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:46.128471 15750 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:02:46.129441 15750 webserver.cc:460] Webserver started at http://127.9.111.67:41061/ using document root <none> and password file <none>
I20260806 17:02:46.130101 15750 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:46.130182 15750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:46.130417 15750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:46.132679 15750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "ce675a50c0ed4b198a765d7ff219c25f"
format_stamp: "Formatted at 2026-08-06 17:02:46 on dist-test-slave-69rg"
server_key: "0d894719ca71c41bef2f1d84ee7c4597"
server_key_iv: "f084ba1b843aa2552c03889d3a1e9e51"
server_key_version: "encryptionkey@0"
I20260806 17:02:46.133311 15750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "ce675a50c0ed4b198a765d7ff219c25f"
format_stamp: "Formatted at 2026-08-06 17:02:46 on dist-test-slave-69rg"
server_key: "0d894719ca71c41bef2f1d84ee7c4597"
server_key_iv: "f084ba1b843aa2552c03889d3a1e9e51"
server_key_version: "encryptionkey@0"
I20260806 17:02:46.137676 15750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260806 17:02:46.140506 15767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:46.142118 15750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:46.142264 15750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "ce675a50c0ed4b198a765d7ff219c25f"
format_stamp: "Formatted at 2026-08-06 17:02:46 on dist-test-slave-69rg"
server_key: "0d894719ca71c41bef2f1d84ee7c4597"
server_key_iv: "f084ba1b843aa2552c03889d3a1e9e51"
server_key_version: "encryptionkey@0"
I20260806 17:02:46.142402 15750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:46.153421 15750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:46.157027 15750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:46.157258 15750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:46.157960 15750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:46.158850 15750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:46.158900 15750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:46.158941 15750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:46.158998 15750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:46.168483 15750 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:36647
I20260806 17:02:46.168504 15882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:36647 every 8 connection(s)
I20260806 17:02:46.169478 15750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:02:46.173298 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 15750
I20260806 17:02:46.173460 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:02:46.173916 9661 external_mini_cluster.cc:1428] Setting key 27a36d33e05bee31c50537aec4566fbd
Aug 06 17:02:46 dist-test-slave-69rg krb5kdc[15367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035766, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:46.184813 15453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:46.171510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:51495 (local address 127.9.111.126:38431)
0806 17:02:46.171658 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:46.171661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:46.172517 (+ 856us) server_negotiation.cc:407] Connection header received
0806 17:02:46.174194 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:46.174198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:46.174261 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:46.174372 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:46.175861 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:46.176521 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:46.177704 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:46.177950 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:46.178034 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:46.180866 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:46.180869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:46.180871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:46.180898 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:46.182619 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:46.183183 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:46.183186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:46.183187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:46.183273 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:46.183727 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:46.183730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:46.183731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:46.183940 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:46.184018 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:46.184569 (+ 551us) server_negotiation.cc:299] Negotiation successful
0806 17:02:46.184682 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 17:02:46.185742 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:46.171888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38431 (local address 127.9.111.67:51495)
0806 17:02:46.172351 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:46.172395 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:46.173918 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:46.174559 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:46.174566 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:46.175064 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:46.175682 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:46.175702 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:46.176642 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:46.176645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:46.177571 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:46.177581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:46.178112 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:46.178115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:46.178262 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:46.178952 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:46.178969 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:46.180668 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:46.182745 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:46.182750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:46.182765 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:46.183051 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:46.183436 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:46.183439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:46.183441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:46.183620 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:46.184037 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:46.184042 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:46.184347 (+ 305us) client_negotiation.cc:769] Sending connection context
0806 17:02:46.184519 (+ 172us) client_negotiation.cc:240] Negotiation successful
0806 17:02:46.184802 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":109,"threads_started":1}
I20260806 17:02:46.187371 15883 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38431
I20260806 17:02:46.187672 15883 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:46.188341 15883 heartbeater.cc:511] Master 127.9.111.126:38431 requested a full tablet report, sending...
I20260806 17:02:46.189563 15412 ts_manager.cc:194] Registered new tserver with Master: ce675a50c0ed4b198a765d7ff219c25f (127.9.111.67:36647)
I20260806 17:02:46.190290 15412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:51495
I20260806 17:02:46.200863 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/test-admin.keytab.
Aug 06 17:02:46 dist-test-slave-69rg krb5kdc[15367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:46.220736 9661 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 17:02:46.223768 15890 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:02:46.222268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44564 (local address 127.9.111.65:37621)
0806 17:02:46.222626 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:46.222635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:46.222677 (+ 42us) server_negotiation.cc:407] Connection header received
0806 17:02:46.222739 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:46.222744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:46.222934 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:46.223083 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:46.223599 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44564: BlockingRecv error: recv got EOF from 127.0.0.1:44564 (error 108)
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":132,"threads_started":1}
W20260806 17:02:46.224076 9661 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/krb5cc' of type FILE
Aug 06 17:02:46 dist-test-slave-69rg krb5kdc[15367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:46.225183 9661 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 17:02:46 dist-test-slave-69rg krb5kdc[15367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:46.226136 9661 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 17:02:46 dist-test-slave-69rg krb5kdc[15367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.65@KRBTEST.COM
I20260806 17:02:46.236374 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:46.227052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.9.111.65:37621)
0806 17:02:46.227250 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:46.227254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:46.227287 (+ 33us) server_negotiation.cc:407] Connection header received
0806 17:02:46.227420 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:46.227422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:46.227463 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:46.227530 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:46.228029 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:46.228907 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:46.229766 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:46.230256 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:46.230352 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:46.232696 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:46.232706 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:46.232725 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:46.232750 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:46.234559 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:46.234967 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:46.234972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:46.234973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:46.235049 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:46.235371 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:46.235374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:46.235375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:46.235624 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:46.235782 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:46.235974 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 17:02:46.236191 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260806 17:02:46.240499 15890 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:02:46.239370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.9.111.65:37621)
0806 17:02:46.239509 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:46.239513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:46.239648 (+ 135us) server_negotiation.cc:407] Connection header received
0806 17:02:46.239799 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:46.239803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:46.239845 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:46.239936 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:46.240393 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44570: BlockingRecv error: recv got EOF from 127.0.0.1:44570 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 17:02:46.240756 9661 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/krb5cc' of type FILE
Aug 06 17:02:46 dist-test-slave-69rg krb5kdc[15367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:46.241945 9661 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 17:02:46 dist-test-slave-69rg krb5kdc[15367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:46.242879 9661 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 17:02:46 dist-test-slave-69rg krb5kdc[15367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.65@KRBTEST.COM
I20260806 17:02:46.252346 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:46.243774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44576 (local address 127.9.111.65:37621)
0806 17:02:46.243931 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:46.243934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:46.243996 (+ 62us) server_negotiation.cc:407] Connection header received
0806 17:02:46.244100 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:46.244102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:46.244140 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:46.244208 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:46.244726 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:46.245285 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:46.246384 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:46.246625 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:46.246724 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:46.249185 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:46.249188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:46.249190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:46.249216 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:46.250660 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:46.251106 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:46.251116 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:46.251118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:46.251192 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:46.251481 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:46.251485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:46.251487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:46.251662 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:46.251843 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:46.252051 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 17:02:46.252172 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260806 17:02:46.255117 15890 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:02:46.254264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44586 (local address 127.9.111.65:37621)
0806 17:02:46.254400 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:46.254404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:46.254462 (+ 58us) server_negotiation.cc:407] Connection header received
0806 17:02:46.254553 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:46.254555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:46.254591 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:46.254656 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:46.255026 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44586: BlockingRecv error: recv got EOF from 127.0.0.1:44586 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 17:02:46.255539 9661 init.cc:244] Error while opening credential cache '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786035732873553-9661-0/krb5kdc/krb5cc)
Aug 06 17:02:46 dist-test-slave-69rg krb5kdc[15367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:46.256613 9661 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 17:02:46 dist-test-slave-69rg krb5kdc[15367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:46.257493 9661 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 17:02:46 dist-test-slave-69rg krb5kdc[15367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.65@KRBTEST.COM
I20260806 17:02:46.266848 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:46.258482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44602 (local address 127.9.111.65:37621)
0806 17:02:46.258603 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:46.258609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:46.258695 (+ 86us) server_negotiation.cc:407] Connection header received
0806 17:02:46.258854 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:46.258857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:46.258893 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:46.258974 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:46.259463 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:46.259952 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:46.260846 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:46.261092 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:46.261201 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:46.263722 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:46.263725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:46.263727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:46.263750 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:46.265095 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:46.265567 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:46.265574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:46.265576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:46.265655 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:46.266000 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:46.266003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:46.266005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:46.266187 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:46.266341 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:46.266568 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 17:02:46.266686 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 17:02:46.267899 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 15477
I20260806 17:02:46.274803 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 15615
I20260806 17:02:46.280438 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 15750
I20260806 17:02:46.286266 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 15380
2026-08-06T17:02:46Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2364 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:02:46 dist-test-slave-69rg krb5kdc[15903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:02:46 dist-test-slave-69rg krb5kdc[15903](info): set up 2 sockets
Aug 06 17:02:46 dist-test-slave-69rg krb5kdc[15903](info): commencing operation
krb5kdc: starting...
W20260806 17:02:47.369869 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:02:47 dist-test-slave-69rg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:02:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.126.keytab.
Entry for principal oryx/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.126.keytab.
I20260806 17:02:47.509799 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:38979
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.126.keytab
--principal=oryx/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:38979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:47.623523 15916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:47.623875 15916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:47.623955 15916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:47.628082 15916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:47.628202 15916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:47.628240 15916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:47.628281 15916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:47.628314 15916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:02:47.633903 15916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:38979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.126.keytab
--principal=oryx/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:38979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:47.635054 15916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:47.636184 15916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 17:02:47.645042 15916 server_base.cc:1034] running on GCE node
W20260806 17:02:47.644958 15924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:47.644919 15922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:47.644929 15921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:47.645995 15916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:47.647130 15916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:47.648324 15916 hybrid_clock.cc:648] HybridClock initialized: now 1786035767648293 us; error 43 us; skew 500 ppm
Aug 06 17:02:47 dist-test-slave-69rg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035767, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:47.651248 15916 init.cc:377] Logged in from keytab as oryx/127.9.111.126@KRBTEST.COM (short username oryx)
I20260806 17:02:47.652169 15916 webserver.cc:460] Webserver started at http://127.9.111.126:34889/ using document root <none> and password file <none>
I20260806 17:02:47.652696 15916 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:47.652755 15916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:47.652930 15916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:47.655055 15916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "aeb0d5145418420593ee76bde8104d59"
format_stamp: "Formatted at 2026-08-06 17:02:47 on dist-test-slave-69rg"
server_key: "5cb969fad3a67a3af2ffd4ae08c80c2a"
server_key_iv: "a65e5c554ccc28bf6955809d4c68cff8"
server_key_version: "encryptionkey@0"
I20260806 17:02:47.655619 15916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "aeb0d5145418420593ee76bde8104d59"
format_stamp: "Formatted at 2026-08-06 17:02:47 on dist-test-slave-69rg"
server_key: "5cb969fad3a67a3af2ffd4ae08c80c2a"
server_key_iv: "a65e5c554ccc28bf6955809d4c68cff8"
server_key_version: "encryptionkey@0"
I20260806 17:02:47.659349 15916 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 17:02:47.661873 15931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:47.663173 15916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 17:02:47.663316 15916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "aeb0d5145418420593ee76bde8104d59"
format_stamp: "Formatted at 2026-08-06 17:02:47 on dist-test-slave-69rg"
server_key: "5cb969fad3a67a3af2ffd4ae08c80c2a"
server_key_iv: "a65e5c554ccc28bf6955809d4c68cff8"
server_key_version: "encryptionkey@0"
I20260806 17:02:47.663439 15916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:47.679312 15916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:47.682688 15916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:47.682909 15916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:47.690348 15916 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:38979
I20260806 17:02:47.690660 15987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:38979 every 8 connection(s)
I20260806 17:02:47.691330 15916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:02:47.694124 15988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:47.697533 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 15916
I20260806 17:02:47.697685 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:02:47.698029 9661 external_mini_cluster.cc:1428] Setting key 769343d0f98c5010d8d5fe8422e22600
I20260806 17:02:47.700240 15988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59: Bootstrap starting.
I20260806 17:02:47.702836 15988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:47.703790 15988 log.cc:824] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:47.706286 15988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59: No bootstrap required, opened a new log
Aug 06 17:02:47 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM
I20260806 17:02:47.710198 15988 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } }
I20260806 17:02:47.710458 15988 raft_consensus.cc:374] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:47.710520 15988 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb0d5145418420593ee76bde8104d59, State: Initialized, Role: FOLLOWER
I20260806 17:02:47.710949 15988 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } }
I20260806 17:02:47.711102 15988 raft_consensus.cc:388] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:02:47.711205 15988 raft_consensus.cc:482] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:02:47.711316 15988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:47.712428 15988 raft_consensus.cc:504] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } }
I20260806 17:02:47.712762 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:47.699481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43918 (local address 127.9.111.126:38979)
0806 17:02:47.699902 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:47.699917 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:47.699953 (+ 36us) server_negotiation.cc:407] Connection header received
0806 17:02:47.700626 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:47.700647 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:47.700969 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:47.701231 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:47.702163 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:47.703235 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:47.704400 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:47.704963 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:47.705082 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:47.707927 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:47.707932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:47.707945 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:47.707972 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:47.709735 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:47.710265 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:47.710270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:47.710275 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:47.710387 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:47.710691 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:47.710694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:47.710695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:47.711164 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:47.711396 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:47.711641 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 17:02:47.711935 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":132,"threads_started":1}
I20260806 17:02:47.712759 15988 leader_election.cc:302] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aeb0d5145418420593ee76bde8104d59; no voters:
I20260806 17:02:47.713168 15988 leader_election.cc:288] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:02:47.713222 15993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:47.713565 15993 raft_consensus.cc:686] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [term 1 LEADER]: Becoming Leader. State: Replica: aeb0d5145418420593ee76bde8104d59, State: Running, Role: LEADER
I20260806 17:02:47.713961 15993 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } }
I20260806 17:02:47.715497 15988 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:47.715705 15994 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aeb0d5145418420593ee76bde8104d59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } }
I20260806 17:02:47.715667 15995 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aeb0d5145418420593ee76bde8104d59. Latest consensus state: current_term: 1 leader_uuid: "aeb0d5145418420593ee76bde8104d59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } }
I20260806 17:02:47.715821 15994 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:47.715821 15995 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:47.716248 15997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:47.719937 15997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:02:47.725711 15997 catalog_manager.cc:1347] Generated new cluster ID: e6a59be1451a40deb161b33abd2f6a28
I20260806 17:02:47.725795 15997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:47.737385 15997 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:02:47.738444 15997 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:47.747761 15997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59: Generated new TSK 0
I20260806 17:02:47.748474 15997 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.65.keytab.
Entry for principal oryx/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.65.keytab.
I20260806 17:02:47.787014 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38979
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.65.keytab
--principal=oryx/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:47.925339 16014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:47.925650 16014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:47.925726 16014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:47.929751 16014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:47.929885 16014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:47.930015 16014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:02:47.935467 16014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.65.keytab
--principal=oryx/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:47.936529 16014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:47.937538 16014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:47.947214 16022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:47.947214 16020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:47.947353 16014 server_base.cc:1034] running on GCE node
W20260806 17:02:47.947211 16019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:47.948007 16014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:47.948702 16014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:47.949900 16014 hybrid_clock.cc:648] HybridClock initialized: now 1786035767949875 us; error 34 us; skew 500 ppm
Aug 06 17:02:47 dist-test-slave-69rg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035767, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:47.952750 16014 init.cc:377] Logged in from keytab as oryx/127.9.111.65@KRBTEST.COM (short username oryx)
I20260806 17:02:47.953701 16014 webserver.cc:460] Webserver started at http://127.9.111.65:39693/ using document root <none> and password file <none>
I20260806 17:02:47.954375 16014 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:47.954473 16014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:47.954691 16014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:47.957062 16014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "4823da74a32c4172a1f08101bc777555"
format_stamp: "Formatted at 2026-08-06 17:02:47 on dist-test-slave-69rg"
server_key: "3dab996652b814ffaba2a82db43e3e49"
server_key_iv: "7385949edabea958ceb99927e3c8369a"
server_key_version: "encryptionkey@0"
I20260806 17:02:47.957922 16014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "4823da74a32c4172a1f08101bc777555"
format_stamp: "Formatted at 2026-08-06 17:02:47 on dist-test-slave-69rg"
server_key: "3dab996652b814ffaba2a82db43e3e49"
server_key_iv: "7385949edabea958ceb99927e3c8369a"
server_key_version: "encryptionkey@0"
I20260806 17:02:47.962007 16014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:02:47.964550 16030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:47.966029 16014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:47.966171 16014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "4823da74a32c4172a1f08101bc777555"
format_stamp: "Formatted at 2026-08-06 17:02:47 on dist-test-slave-69rg"
server_key: "3dab996652b814ffaba2a82db43e3e49"
server_key_iv: "7385949edabea958ceb99927e3c8369a"
server_key_version: "encryptionkey@0"
I20260806 17:02:47.966306 16014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:47.988637 16014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:47.992534 16014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:47.992749 16014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:47.993433 16014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:47.994294 16014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:47.994349 16014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:47.994387 16014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:47.994402 16014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:48.003764 16014 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:35221
I20260806 17:02:48.003788 16145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:35221 every 8 connection(s)
I20260806 17:02:48.004729 16014 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:02:48.005584 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 16014
I20260806 17:02:48.005716 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:02:48.006129 9661 external_mini_cluster.cc:1428] Setting key 1781b34c78923ed5818882079e141463
Aug 06 17:02:48 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035767, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.9.111.66@KRBTEST.COM; defaulting to no policy
I20260806 17:02:48.020237 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:48.006976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:45311 (local address 127.9.111.126:38979)
0806 17:02:48.007137 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:48.007142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:48.008354 (+ 1212us) server_negotiation.cc:407] Connection header received
0806 17:02:48.009382 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:48.009387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:48.009443 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:48.009541 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:48.010833 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.011397 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.012595 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.012846 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.012960 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:48.016184 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:48.016191 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:48.016195 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:48.016232 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:48.017999 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.018580 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.018584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.018586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.018683 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.019050 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.019054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.019056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.019267 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:48.019379 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:48.019915 (+ 536us) server_negotiation.cc:299] Negotiation successful
0806 17:02:48.020061 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.111.66@KRBTEST.COM" created.
I20260806 17:02:48.021138 16148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:48.007489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.65:45311)
0806 17:02:48.008198 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:48.008241 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:48.009141 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:48.009691 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:48.009697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:48.010139 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:48.010673 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:48.010688 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.011541 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.011546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:48.012464 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:48.012474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.013019 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.013023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:48.013206 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:48.013953 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:48.013981 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:48.015909 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:48.018158 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:48.018164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:48.018182 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:48.018462 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:48.018781 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:48.018785 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:48.018787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:48.018948 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:48.019380 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:48.019388 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:48.019667 (+ 279us) client_negotiation.cc:769] Sending connection context
0806 17:02:48.019873 (+ 206us) client_negotiation.cc:240] Negotiation successful
0806 17:02:48.020243 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":118,"threads_started":1}
I20260806 17:02:48.022521 16146 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38979
I20260806 17:02:48.022802 16146 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:48.023330 16146 heartbeater.cc:511] Master 127.9.111.126:38979 requested a full tablet report, sending...
I20260806 17:02:48.024869 15947 ts_manager.cc:194] Registered new tserver with Master: 4823da74a32c4172a1f08101bc777555 (127.9.111.65:35221)
I20260806 17:02:48.026130 15947 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:45311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.66.keytab.
Entry for principal oryx/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.66.keytab.
I20260806 17:02:48.036962 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38979
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.66.keytab
--principal=oryx/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:48.163219 16152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:48.163506 16152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:48.163554 16152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:48.167829 16152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:48.167898 16152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:48.167961 16152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:02:48.173406 16152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.66.keytab
--principal=oryx/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:48.174587 16152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:48.175571 16152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:48.185144 16160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:48.185199 16157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:48.185148 16158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:48.185550 16152 server_base.cc:1034] running on GCE node
I20260806 17:02:48.186033 16152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:48.186728 16152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:48.187924 16152 hybrid_clock.cc:648] HybridClock initialized: now 1786035768187908 us; error 26 us; skew 500 ppm
Aug 06 17:02:48 dist-test-slave-69rg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035768, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:48.190984 16152 init.cc:377] Logged in from keytab as oryx/127.9.111.66@KRBTEST.COM (short username oryx)
I20260806 17:02:48.191911 16152 webserver.cc:460] Webserver started at http://127.9.111.66:42435/ using document root <none> and password file <none>
I20260806 17:02:48.192514 16152 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:48.192608 16152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:48.192831 16152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:48.195019 16152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "fe05064e06e544d48bd43c56811ceca1"
format_stamp: "Formatted at 2026-08-06 17:02:48 on dist-test-slave-69rg"
server_key: "281757a5931364491d816a1ae388284c"
server_key_iv: "07100954a941df9292d12c0142bded75"
server_key_version: "encryptionkey@0"
I20260806 17:02:48.195608 16152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "fe05064e06e544d48bd43c56811ceca1"
format_stamp: "Formatted at 2026-08-06 17:02:48 on dist-test-slave-69rg"
server_key: "281757a5931364491d816a1ae388284c"
server_key_iv: "07100954a941df9292d12c0142bded75"
server_key_version: "encryptionkey@0"
I20260806 17:02:48.199352 16152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:02:48.201786 16168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:48.203110 16152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:02:48.203239 16152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "fe05064e06e544d48bd43c56811ceca1"
format_stamp: "Formatted at 2026-08-06 17:02:48 on dist-test-slave-69rg"
server_key: "281757a5931364491d816a1ae388284c"
server_key_iv: "07100954a941df9292d12c0142bded75"
server_key_version: "encryptionkey@0"
I20260806 17:02:48.203356 16152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:48.229702 16152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:48.233033 16152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:48.233249 16152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:48.233924 16152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:48.234794 16152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:48.234843 16152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:48.234882 16152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:48.234939 16152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:48.244412 16152 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:34495
I20260806 17:02:48.244437 16285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:34495 every 8 connection(s)
I20260806 17:02:48.245383 16152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
Aug 06 17:02:48 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035768, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM
I20260806 17:02:48.255538 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 16152
I20260806 17:02:48.255661 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:02:48.255935 9661 external_mini_cluster.cc:1428] Setting key 023d7d8fb9394e6337ab4030c9a20266
I20260806 17:02:48.259442 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:48.247315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:57187 (local address 127.9.111.126:38979)
0806 17:02:48.247463 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:48.247467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:48.248661 (+ 1194us) server_negotiation.cc:407] Connection header received
0806 17:02:48.249618 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:48.249621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:48.249681 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:48.249788 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:48.250996 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.251548 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.252704 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.252941 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.253059 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:48.255759 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:48.255762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:48.255764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:48.255795 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:48.257296 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.257868 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.257872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.257874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.257959 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.258336 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.258339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.258341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.258532 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:48.258644 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:48.259144 (+ 500us) server_negotiation.cc:299] Negotiation successful
0806 17:02:48.259280 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 17:02:48.260299 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:48.247761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.66:57187)
0806 17:02:48.248504 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:48.248547 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:48.249409 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:48.249983 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:48.249989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:48.250357 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:48.250846 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:48.250860 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.251697 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.251700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:48.252559 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:48.252568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.253102 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.253107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:48.253279 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:48.253922 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:48.253939 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:48.255506 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:48.257432 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:48.257438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:48.257453 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:48.257728 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:48.258055 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:48.258059 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:48.258061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:48.258222 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:48.258643 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:48.258649 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:48.258905 (+ 256us) client_negotiation.cc:769] Sending connection context
0806 17:02:48.259100 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 17:02:48.259445 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":104,"threads_started":1}
I20260806 17:02:48.261595 16286 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38979
I20260806 17:02:48.261897 16286 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:48.262562 16286 heartbeater.cc:511] Master 127.9.111.126:38979 requested a full tablet report, sending...
I20260806 17:02:48.263753 15947 ts_manager.cc:194] Registered new tserver with Master: fe05064e06e544d48bd43c56811ceca1 (127.9.111.66:34495)
I20260806 17:02:48.264360 15947 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:57187
WARNING: no policy specified for oryx/127.9.111.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.111.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.67.keytab.
Entry for principal oryx/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.67.keytab.
I20260806 17:02:48.287037 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38979
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.67.keytab
--principal=oryx/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:48.401554 16292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:48.401904 16292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:48.401957 16292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:48.405968 16292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:48.406028 16292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:48.406086 16292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:02:48.411580 16292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.67.keytab
--principal=oryx/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:48.412658 16292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:48.413658 16292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:48.423748 16300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:48.423753 16297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:48.423806 16298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:48.424153 16292 server_base.cc:1034] running on GCE node
I20260806 17:02:48.424638 16292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:48.425403 16292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:48.426638 16292 hybrid_clock.cc:648] HybridClock initialized: now 1786035768426603 us; error 44 us; skew 500 ppm
Aug 06 17:02:48 dist-test-slave-69rg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035768, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:48.429531 16292 init.cc:377] Logged in from keytab as oryx/127.9.111.67@KRBTEST.COM (short username oryx)
I20260806 17:02:48.430522 16292 webserver.cc:460] Webserver started at http://127.9.111.67:42349/ using document root <none> and password file <none>
I20260806 17:02:48.431114 16292 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:48.431222 16292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:48.431463 16292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:48.433504 16292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "c84fd741df5e4071835c818d2e769a0d"
format_stamp: "Formatted at 2026-08-06 17:02:48 on dist-test-slave-69rg"
server_key: "311289b67cff36e881196945ceb230b3"
server_key_iv: "4e930eeaa68cb2ebffd77bddffd3b591"
server_key_version: "encryptionkey@0"
I20260806 17:02:48.434113 16292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "c84fd741df5e4071835c818d2e769a0d"
format_stamp: "Formatted at 2026-08-06 17:02:48 on dist-test-slave-69rg"
server_key: "311289b67cff36e881196945ceb230b3"
server_key_iv: "4e930eeaa68cb2ebffd77bddffd3b591"
server_key_version: "encryptionkey@0"
I20260806 17:02:48.438273 16292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 17:02:48.440636 16307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:48.441949 16292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:02:48.442080 16292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "c84fd741df5e4071835c818d2e769a0d"
format_stamp: "Formatted at 2026-08-06 17:02:48 on dist-test-slave-69rg"
server_key: "311289b67cff36e881196945ceb230b3"
server_key_iv: "4e930eeaa68cb2ebffd77bddffd3b591"
server_key_version: "encryptionkey@0"
I20260806 17:02:48.442240 16292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:48.477666 16292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:48.480798 16292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:48.481037 16292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:48.481714 16292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:48.482612 16292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:48.482689 16292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:48.482764 16292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:48.482810 16292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:48.491894 16292 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:33877
I20260806 17:02:48.491910 16423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:33877 every 8 connection(s)
I20260806 17:02:48.492878 16292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:02:48.494918 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 16292
I20260806 17:02:48.495074 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:02:48.495446 9661 external_mini_cluster.cc:1428] Setting key 1b38a39c56d51cc2ab33436fe4981a99
Aug 06 17:02:48 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035768, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM
I20260806 17:02:48.507431 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:48.495014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:35075 (local address 127.9.111.126:38979)
0806 17:02:48.495179 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:48.495184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:48.496323 (+ 1139us) server_negotiation.cc:407] Connection header received
0806 17:02:48.497398 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:48.497401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:48.497458 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:48.497550 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:48.498876 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.499452 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.500565 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.500785 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.500875 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:48.503570 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:48.503574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:48.503576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:48.503604 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:48.505272 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.505849 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.505853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.505855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.505961 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.506311 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.506314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.506315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.506521 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:48.506724 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:48.507168 (+ 444us) server_negotiation.cc:299] Negotiation successful
0806 17:02:48.507283 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 17:02:48.508442 16426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:48.495390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.67:35075)
0806 17:02:48.496114 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:48.496223 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:48.497190 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:48.497749 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:48.497756 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:48.498233 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:48.498734 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:48.498748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.499567 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.499569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:48.500432 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:48.500440 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.500921 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.500924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:48.501063 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:48.501676 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:48.501694 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:48.503376 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:48.505408 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:48.505413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:48.505428 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:48.505726 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:48.506043 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:48.506045 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:48.506046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:48.506198 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:48.506658 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:48.506669 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:48.506942 (+ 273us) client_negotiation.cc:769] Sending connection context
0806 17:02:48.507110 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 17:02:48.507435 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":144,"threads_started":1}
I20260806 17:02:48.509949 16424 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38979
I20260806 17:02:48.510228 16424 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:48.510857 16424 heartbeater.cc:511] Master 127.9.111.126:38979 requested a full tablet report, sending...
I20260806 17:02:48.512127 15947 ts_manager.cc:194] Registered new tserver with Master: c84fd741df5e4071835c818d2e769a0d (127.9.111.67:33877)
I20260806 17:02:48.512758 15947 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:35075
I20260806 17:02:48.522162 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.111.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.111.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.111.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.125.keytab.
Entry for principal oryx/127.9.111.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.125.keytab.
I20260806 17:02:48.553581 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.125:43363
--webserver_interface=127.9.111.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.125.keytab
--principal=oryx/127.9.111.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:38979,127.9.111.125:43363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:48.677145 16429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:48.677457 16429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:48.677500 16429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:48.681844 16429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:48.681918 16429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:48.681939 16429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:48.681957 16429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:48.681973 16429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:02:48.687492 16429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:38979,127.9.111.125:43363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.125.keytab
--principal=oryx/127.9.111.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.125:43363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.111.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
W20260806 17:02:48.687752 16429 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.9.111.126:38979,127.9.111.125:43363'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 17:02:48.688697 16429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:48.689776 16429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:48.699280 16434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:48.699280 16435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:48.699442 16429 server_base.cc:1034] running on GCE node
W20260806 17:02:48.699595 16437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:48.700181 16429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:48.701498 16429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:48.702684 16429 hybrid_clock.cc:648] HybridClock initialized: now 1786035768702638 us; error 56 us; skew 500 ppm
Aug 06 17:02:48 dist-test-slave-69rg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035768, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:48.705911 16429 init.cc:377] Logged in from keytab as oryx/127.9.111.125@KRBTEST.COM (short username oryx)
I20260806 17:02:48.706784 16429 webserver.cc:460] Webserver started at http://127.9.111.125:38773/ using document root <none> and password file <none>
I20260806 17:02:48.707449 16429 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:48.707556 16429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:48.707788 16429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:48.709911 16429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/data/instance:
uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4"
format_stamp: "Formatted at 2026-08-06 17:02:48 on dist-test-slave-69rg"
server_key: "cb18e49820104613c788ad0233504820"
server_key_iv: "8fc6d09b3abe00f076b093a4ded7eff1"
server_key_version: "encryptionkey@0"
I20260806 17:02:48.710572 16429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/wal/instance:
uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4"
format_stamp: "Formatted at 2026-08-06 17:02:48 on dist-test-slave-69rg"
server_key: "cb18e49820104613c788ad0233504820"
server_key_iv: "8fc6d09b3abe00f076b093a4ded7eff1"
server_key_version: "encryptionkey@0"
I20260806 17:02:48.714561 16429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 17:02:48.717527 16445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:48.719100 16429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 17:02:48.719290 16429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/wal
uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4"
format_stamp: "Formatted at 2026-08-06 17:02:48 on dist-test-slave-69rg"
server_key: "cb18e49820104613c788ad0233504820"
server_key_iv: "8fc6d09b3abe00f076b093a4ded7eff1"
server_key_version: "encryptionkey@0"
I20260806 17:02:48.719444 16429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:48.731813 16429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:48.735150 16429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:48.735370 16429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:48.743036 16429 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.125:43363
I20260806 17:02:48.743335 16497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.125:43363 every 8 connection(s)
I20260806 17:02:48.744136 16429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/data/info.pb
I20260806 17:02:48.747197 16498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:48.751204 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 16429
I20260806 17:02:48.751159 16498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 17:02:48 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035768, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.125@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM
Aug 06 17:02:48 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.111.125@KRBTEST.COM
I20260806 17:02:48.764531 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:48.753117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.9.111.126:38979)
0806 17:02:48.753273 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:48.753278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:48.753580 (+ 302us) server_negotiation.cc:407] Connection header received
0806 17:02:48.754421 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:48.754424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:48.754491 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:48.754581 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:48.755447 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.755988 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.757029 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.757257 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.757381 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:48.760200 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:48.760207 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:48.760211 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:48.760246 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:48.761960 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.762503 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.762506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.762507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.762589 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.762979 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.762985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.762989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.763332 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:48.763457 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:48.764189 (+ 732us) server_negotiation.cc:299] Negotiation successful
0806 17:02:48.764335 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 17:02:48.765614 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:48.752816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51612 (local address 127.9.111.125:43363)
0806 17:02:48.753268 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:48.753285 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:48.753342 (+ 57us) server_negotiation.cc:407] Connection header received
0806 17:02:48.754137 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:48.754154 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:48.754344 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:48.754525 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:48.755368 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.756311 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.757139 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.757511 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.757627 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:48.760625 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:48.760630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:48.760662 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:48.760691 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:48.762691 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.763211 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.763216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.763217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.763303 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.763618 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.763622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.763624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.763931 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:48.764212 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:48.764407 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 17:02:48.764535 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":174,"threads_started":1}
I20260806 17:02:48.765621 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:48.753006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.0.0.1:43926)
0806 17:02:48.753445 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:48.753483 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:48.754212 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:48.754633 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:48.754638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:48.755044 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:48.755296 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:48.755310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.756130 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.756134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:48.756900 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:48.756909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.757413 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.757417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:48.757574 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:48.758246 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:48.758263 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:48.759960 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:48.762066 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:48.762076 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:48.762089 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:48.762401 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:48.762682 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:48.762685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:48.762687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:48.762871 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:48.763491 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:48.763497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:48.763868 (+ 371us) client_negotiation.cc:769] Sending connection context
0806 17:02:48.764058 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 17:02:48.764429 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":157,"threads_started":1}
I20260806 17:02:48.768317 16498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 17:02:48 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035768, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.125@KRBTEST.COM for oryx/127.9.111.125@KRBTEST.COM
I20260806 17:02:48.777267 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:48.768929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:43363 (local address 127.0.0.1:51618)
0806 17:02:48.769029 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:48.769045 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:48.769121 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:48.769333 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:48.769336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:48.769479 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:48.769574 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:48.769579 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.770418 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.770420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:48.771073 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:48.771082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.772219 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.772222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:48.772297 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:48.772848 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:48.772860 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:48.774290 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:48.775945 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:48.775948 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:48.775949 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:48.776130 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:48.776429 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:48.776431 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:48.776432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:48.776504 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:48.776879 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:48.776881 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:48.776957 (+ 76us) client_negotiation.cc:769] Sending connection context
0806 17:02:48.777037 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 17:02:48.777139 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 17:02:48.777424 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:48.769004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.9.111.125:43363)
0806 17:02:48.769129 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:48.769133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:48.769149 (+ 16us) server_negotiation.cc:407] Connection header received
0806 17:02:48.769186 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:48.769188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:48.769222 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:48.769306 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:48.769700 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.770310 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.771248 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.772101 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.772204 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:48.774438 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:48.774441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:48.774443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:48.774465 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:48.775843 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.776254 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.776257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.776258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.776339 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.776593 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.776599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.776600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.776797 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:48.776880 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:48.777125 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 17:02:48.777227 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 17:02:48.780207 16498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: Bootstrap starting.
I20260806 17:02:48.782648 16498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:48.783385 16498 log.cc:824] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:48.785408 16498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: No bootstrap required, opened a new log
I20260806 17:02:48.788739 16498 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } }
I20260806 17:02:48.788996 16498 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:48.789093 16498 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d380aa2fee4c3b9eb65c7a0d9ebfe4, State: Initialized, Role: FOLLOWER
I20260806 17:02:48.789567 16498 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } }
I20260806 17:02:48.790129 16504 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } } }
I20260806 17:02:48.790299 16504 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:02:48.790906 16498 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 17:02:48.795825 16516 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:02:48.795914 16516 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:02:48.798381 16429 master_runner.cc:428] Detected that this master b5d380aa2fee4c3b9eb65c7a0d9ebfe4 is joining an existing cluster
I20260806 17:02:48.798452 16429 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.111.125:43363
I20260806 17:02:48.811609 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:48.801253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.9.111.126:38979)
0806 17:02:48.801419 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:48.801425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:48.801731 (+ 306us) server_negotiation.cc:407] Connection header received
0806 17:02:48.802002 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:48.802006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:48.802070 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:48.802163 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:48.802943 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.803789 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.805013 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.805271 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.805402 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:48.807593 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:48.807598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:48.807601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:48.807634 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:48.809614 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.810290 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.810294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.810296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.810394 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.810672 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.810676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.810678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.810928 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:48.811039 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:48.811323 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 17:02:48.811457 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 17:02:48.811609 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:48.801678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51632 (local address 127.9.111.125:43363)
0806 17:02:48.801780 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:48.801784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:48.802241 (+ 457us) server_negotiation.cc:407] Connection header received
0806 17:02:48.802409 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:48.802412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:48.802455 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:48.802533 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:48.803059 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.803587 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.804901 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.805250 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.805361 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:48.807716 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:48.807719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:48.807722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:48.807746 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:48.809596 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.810203 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.810210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.810212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.810295 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.810624 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.810628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.810630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.810831 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:48.810940 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:48.811258 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 17:02:48.811414 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":42}
I20260806 17:02:48.811607 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:48.801550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:43363 (local address 127.0.0.1:51632)
0806 17:02:48.802046 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:48.802058 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:48.802297 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:48.802577 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:48.802582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:48.802795 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:48.802940 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:48.802946 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.803719 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.803722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:48.804764 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:48.804776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.805402 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.805405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:48.805507 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:48.806850 (+ 1343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:48.806866 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:48.807564 (+ 698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:48.809731 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:48.809734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:48.809736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:48.810095 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:48.810394 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:48.810397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:48.810400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:48.810495 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:48.810940 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:48.810943 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:48.811053 (+ 110us) client_negotiation.cc:769] Sending connection context
0806 17:02:48.811162 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 17:02:48.811343 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":264,"threads_started":1}
I20260806 17:02:48.812240 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:48.801141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.0.0.1:43936)
0806 17:02:48.801449 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:48.801473 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:48.801767 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:48.802243 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:48.802247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:48.802552 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:48.802802 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:48.802812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.803903 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.803906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:48.804893 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:48.804901 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.805532 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.805538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:48.805632 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:48.806397 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:48.806417 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:48.807290 (+ 873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:48.809770 (+ 2480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:48.809773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:48.809776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:48.810151 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:48.810475 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:48.810482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:48.810484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:48.810572 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:48.811027 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:48.811030 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:48.811099 (+ 69us) client_negotiation.cc:769] Sending connection context
0806 17:02:48.811890 (+ 791us) client_negotiation.cc:240] Negotiation successful
0806 17:02:48.812033 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":25,"thread_start_us":103,"threads_started":1}
Aug 06 17:02:48 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035767, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.126@KRBTEST.COM for oryx/127.9.111.125@KRBTEST.COM
I20260806 17:02:48.828723 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:48.816993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:43363 (local address 127.0.0.1:51640)
0806 17:02:48.817443 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:48.817501 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:48.817661 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:48.818028 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:48.818033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:48.818416 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:48.818623 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:48.818634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.819577 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.819582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:48.820407 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:48.820423 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.820935 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.820940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:48.821085 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:48.821751 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:48.821775 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:48.824007 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:48.826554 (+ 2547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:48.826566 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:48.826572 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:48.826918 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:48.827294 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:48.827298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:48.827300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:48.827405 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:48.827941 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:48.827949 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:48.828102 (+ 153us) client_negotiation.cc:769] Sending connection context
0806 17:02:48.828279 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 17:02:48.828506 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":153,"threads_started":1}
I20260806 17:02:48.828763 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:48.817020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.9.111.125:43363)
0806 17:02:48.817175 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:48.817180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:48.817594 (+ 414us) server_negotiation.cc:407] Connection header received
0806 17:02:48.817836 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:48.817841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:48.817901 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:48.817987 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:48.818738 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.819454 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.820561 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.820807 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.820920 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:48.824200 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:48.824208 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:48.824214 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:48.824252 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:48.826400 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.827042 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.827049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.827054 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.827172 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.827520 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.827528 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.827531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.827809 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:48.827928 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:48.828376 (+ 448us) server_negotiation.cc:299] Negotiation successful
0806 17:02:48.828528 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
WARNING: no policy specified for oryx/127.9.111.124@KRBTEST.COM; defaulting to no policy
I20260806 17:02:48.831328 15947 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.111.125:43363: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4"
member_type: NON_VOTER
last_known_addr {
host: "127.9.111.125"
port: 43363
}
attrs {
promote: true
}
}
dest_uuid: "aeb0d5145418420593ee76bde8104d59"
cas_config_opid_index: -1
I20260806 17:02:48.831993 15947 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: NON_VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.111.124@KRBTEST.COM" created.
I20260806 17:02:48.834050 16527 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b5d380aa2fee4c3b9eb65c7a0d9ebfe4 (127.9.111.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: NON_VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: true } } }
I20260806 17:02:48.834818 16472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:48.835834 16527 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.111.125:43363
I20260806 17:02:48.836179 16527 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aeb0d5145418420593ee76bde8104d59" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: NON_VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: true } } }
I20260806 17:02:48.836298 16527 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:48.836553 16429 master.cc:537] Master@127.9.111.125:43363 shutting down...
I20260806 17:02:48.836694 16472 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 1 FOLLOWER]: Refusing update from remote peer aeb0d5145418420593ee76bde8104d59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 17:02:48.837903 16527 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: NON_VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 17:02:48.840365 15933 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260806 17:02:48.841943 16429 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:48.842257 16429 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 17:02:48.842304 15933 consensus_peers.cc:487] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 -> Peer b5d380aa2fee4c3b9eb65c7a0d9ebfe4 (127.9.111.125:43363): Couldn't send request to peer b5d380aa2fee4c3b9eb65c7a0d9ebfe4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 17:02:48.842366 16429 tablet_replica.cc:330] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: stopping tablet replica
I20260806 17:02:48.848708 16429 master.cc:552] Master@127.9.111.125:43363 shutdown complete.
I20260806 17:02:48.848807 16429 master_runner.cc:305] Clearing existing system tablet
I20260806 17:02:48.849068 16429 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:48.849128 16429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:48.850659 16429 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 17:02:48.851909 16533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:48.852329 16429 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:48.852505 16429 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 17:02:48.852567 16429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/wal
uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4"
format_stamp: "Formatted at 2026-08-06 17:02:48 on dist-test-slave-69rg"
server_key: "cb18e49820104613c788ad0233504820"
server_key_iv: "8fc6d09b3abe00f076b093a4ded7eff1"
server_key_version: "encryptionkey@0"
I20260806 17:02:48.854774 16429 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.111.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.124.keytab.
Entry for principal oryx/127.9.111.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.124.keytab.
I20260806 17:02:48.857455 16429 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 17:02:48.857564 16429 log.cc:1196] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 17:02:48.857909 16429 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: Deleting consensus metadata
I20260806 17:02:48.858142 16429 master_runner.cc:315] Copying system tablet from 127.9.111.126:38979
I20260806 17:02:48.858445 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.124:46063
--webserver_interface=127.9.111.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.124.keytab
--principal=oryx/127.9.111.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:38979,127.9.111.125:43363,127.9.111.124:46063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:02:48.859436 16429 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: tablet copy: Beginning tablet copy session from remote peer at address 127.9.111.126:38979
I20260806 17:02:48.869092 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:48.860576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43948 (local address 127.9.111.126:38979)
0806 17:02:48.860778 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:48.860786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:48.860804 (+ 18us) server_negotiation.cc:407] Connection header received
0806 17:02:48.860859 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:48.860865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:48.860924 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:48.861023 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:48.861649 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.862548 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.863485 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.863739 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.863866 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:48.865346 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:48.865350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:48.865353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:48.865387 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:48.867167 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.867670 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.867674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.867676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.867770 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:48.868164 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:48.868167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:48.868169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:48.868374 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:48.868494 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:48.868780 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 17:02:48.868905 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 17:02:48.869120 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:48.860071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.0.0.1:43948)
0806 17:02:48.860357 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:48.860373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:48.860484 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:48.861082 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:48.861086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:48.861341 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:48.861481 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:48.861490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.862690 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.862697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:48.863335 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:48.863346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:48.863873 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:48.863877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:48.863992 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:48.864784 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:48.864797 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:48.865181 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:48.867288 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:48.867293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:48.867295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:48.867541 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:48.867921 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:48.867925 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:48.867927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:48.868059 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:48.868482 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:48.868486 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:48.868600 (+ 114us) client_negotiation.cc:769] Sending connection context
0806 17:02:48.868720 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 17:02:48.868890 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":118,"threads_started":1}
I20260806 17:02:48.870289 15972 tablet_copy_service.cc:140] P aeb0d5145418420593ee76bde8104d59: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b5d380aa2fee4c3b9eb65c7a0d9ebfe4 ({username='oryx', principal='oryx/127.9.111.125@KRBTEST.COM'} at 127.0.0.1:43948)
I20260806 17:02:48.870503 15972 tablet_copy_service.cc:161] P aeb0d5145418420593ee76bde8104d59: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b5d380aa2fee4c3b9eb65c7a0d9ebfe4 at {username='oryx', principal='oryx/127.9.111.125@KRBTEST.COM'} at 127.0.0.1:43948: session id = b5d380aa2fee4c3b9eb65c7a0d9ebfe4-00000000000000000000000000000000
I20260806 17:02:48.872486 15972 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 17:02:48.874169 16429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 17:02:48.876245 16429 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: tablet copy: Starting download of 0 data blocks...
I20260806 17:02:48.876471 16429 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: tablet copy: Starting download of 1 WAL segments...
I20260806 17:02:48.878300 16429 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 17:02:48.879968 15972 tablet_copy_service.cc:342] P aeb0d5145418420593ee76bde8104d59: Request end of tablet copy session b5d380aa2fee4c3b9eb65c7a0d9ebfe4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.111.125@KRBTEST.COM'} at 127.0.0.1:43948
I20260806 17:02:48.880100 15972 tablet_copy_service.cc:433] P aeb0d5145418420593ee76bde8104d59: ending tablet copy session b5d380aa2fee4c3b9eb65c7a0d9ebfe4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b5d380aa2fee4c3b9eb65c7a0d9ebfe4
I20260806 17:02:48.886940 16429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:48.888666 16541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:48.888760 16544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:48.889456 16542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:48.889533 16429 server_base.cc:1034] running on GCE node
I20260806 17:02:48.889882 16429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:48.890239 16429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:48.891405 16429 hybrid_clock.cc:648] HybridClock initialized: now 1786035768891415 us; error 48 us; skew 500 ppm
Aug 06 17:02:48 dist-test-slave-69rg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035768, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:48.893059 16429 init.cc:377] Logged in from keytab as oryx/127.9.111.125@KRBTEST.COM (short username oryx)
I20260806 17:02:48.893617 16429 webserver.cc:460] Webserver started at http://127.9.111.125:41015/ using document root <none> and password file <none>
I20260806 17:02:48.893932 16429 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:48.893990 16429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:48.898276 16429 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.001s
I20260806 17:02:48.899427 16551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:48.899653 16429 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 17:02:48.899757 16429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/wal
uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4"
format_stamp: "Formatted at 2026-08-06 17:02:48 on dist-test-slave-69rg"
server_key: "cb18e49820104613c788ad0233504820"
server_key_iv: "8fc6d09b3abe00f076b093a4ded7eff1"
server_key_version: "encryptionkey@0"
I20260806 17:02:48.899852 16429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:48.909148 16429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:48.911310 16429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:48.916530 16429 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.125:43363
I20260806 17:02:48.917205 16429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-1/data/info.pb
I20260806 17:02:48.917598 16603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.125:43363 every 8 connection(s)
I20260806 17:02:48.918885 16604 sys_catalog.cc:263] Verifying existing consensus state
I20260806 17:02:48.919706 16604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: Bootstrap starting.
I20260806 17:02:48.928546 16604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 17:02:48.928925 16604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: Bootstrap complete.
I20260806 17:02:48.929641 16604 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: NON_VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: true } }
I20260806 17:02:48.929832 16604 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b5d380aa2fee4c3b9eb65c7a0d9ebfe4, State: Initialized, Role: LEARNER
I20260806 17:02:48.929975 16604 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: NON_VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: true } }
I20260806 17:02:48.930547 16604 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:48.933894 16618 catalog_manager.cc:1259] Loaded cluster ID: e6a59be1451a40deb161b33abd2f6a28
I20260806 17:02:48.933952 16618 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: loading cluster ID for follower catalog manager: success
I20260806 17:02:48.936302 16618 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: acquiring CA information for follower catalog manager: success
I20260806 17:02:48.938145 16618 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 17:02:48.941967 16607 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: NON_VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: true } } }
I20260806 17:02:48.942117 16607 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [sys.catalog]: This master's current role is: LEARNER
W20260806 17:02:49.021653 16539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:49.021937 16539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:49.022025 16539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:49.026795 16539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:49.026865 16539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:49.026885 16539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:49.026902 16539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:49.026957 16539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:02:49.028383 16146 heartbeater.cc:503] Master 127.9.111.126:38979 was elected leader, sending a full tablet report...
I20260806 17:02:49.032968 16539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:38979,127.9.111.125:43363,127.9.111.124:46063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.124.keytab
--principal=oryx/127.9.111.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.124:46063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.111.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:49.034174 16539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:49.035259 16539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:49.044466 16624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:49.044467 16623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:49.044592 16626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:49.045006 16539 server_base.cc:1034] running on GCE node
I20260806 17:02:49.045919 16539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:49.047283 16539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:49.048543 16539 hybrid_clock.cc:648] HybridClock initialized: now 1786035769048530 us; error 40 us; skew 500 ppm
Aug 06 17:02:49 dist-test-slave-69rg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035769, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:49.051692 16539 init.cc:377] Logged in from keytab as oryx/127.9.111.124@KRBTEST.COM (short username oryx)
I20260806 17:02:49.052623 16539 webserver.cc:460] Webserver started at http://127.9.111.124:37559/ using document root <none> and password file <none>
I20260806 17:02:49.053180 16539 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:49.053251 16539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:49.053426 16539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:49.055611 16539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/data/instance:
uuid: "1140d9043c1a451a99d278b0cb63d384"
format_stamp: "Formatted at 2026-08-06 17:02:49 on dist-test-slave-69rg"
server_key: "91a30a59c68af399b78d9c9cb4d693d8"
server_key_iv: "6fecb98ab8b29a4785e052171dff2626"
server_key_version: "encryptionkey@0"
I20260806 17:02:49.056146 16539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/wal/instance:
uuid: "1140d9043c1a451a99d278b0cb63d384"
format_stamp: "Formatted at 2026-08-06 17:02:49 on dist-test-slave-69rg"
server_key: "91a30a59c68af399b78d9c9cb4d693d8"
server_key_iv: "6fecb98ab8b29a4785e052171dff2626"
server_key_version: "encryptionkey@0"
I20260806 17:02:49.060146 16539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 17:02:49.062784 16634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:49.064291 16539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 17:02:49.064417 16539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/wal
uuid: "1140d9043c1a451a99d278b0cb63d384"
format_stamp: "Formatted at 2026-08-06 17:02:49 on dist-test-slave-69rg"
server_key: "91a30a59c68af399b78d9c9cb4d693d8"
server_key_iv: "6fecb98ab8b29a4785e052171dff2626"
server_key_version: "encryptionkey@0"
I20260806 17:02:49.064522 16539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:49.077654 16539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:49.081562 16539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:49.081768 16539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:49.091962 16539 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.124:46063
I20260806 17:02:49.092356 16688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.124:46063 every 8 connection(s)
I20260806 17:02:49.093116 16539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/data/info.pb
I20260806 17:02:49.096407 16689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:49.098572 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 16539
I20260806 17:02:49.101593 16689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 17:02:49 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.111.124@KRBTEST.COM
Aug 06 17:02:49 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035769, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.124@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM
I20260806 17:02:49.114838 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.103942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43960 (local address 127.9.111.126:38979)
0806 17:02:49.104089 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.104094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.104422 (+ 328us) server_negotiation.cc:407] Connection header received
0806 17:02:49.104615 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.104625 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.104672 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.104764 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.105629 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.106264 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.107698 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.107960 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.108084 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.110561 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.110568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.110573 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.110607 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.112442 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.113012 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.113015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.113017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.113108 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.113455 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.113458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.113460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.113693 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.113841 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.114482 (+ 641us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.114621 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 17:02:49.115648 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.100214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59178 (local address 127.9.111.124:46063)
0806 17:02:49.100749 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.100767 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.100809 (+ 42us) server_negotiation.cc:407] Connection header received
0806 17:02:49.101646 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.101668 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.102160 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.102601 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.103678 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.105060 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.106208 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.106909 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.107024 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.110032 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.110039 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.110050 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.110074 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.112343 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.112910 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.112914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.112916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.113082 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.113429 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.113433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.113437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.113964 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.114122 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.114496 (+ 374us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.114736 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":377,"thread_start_us":164,"threads_started":1}
I20260806 17:02:49.115640 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.103579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.0.0.1:43960)
0806 17:02:49.104173 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.104215 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.104489 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.104819 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.104824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.105290 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:49.105472 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.105480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.106405 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.106408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.107500 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.107510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.108127 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.108131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.108245 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.108809 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:49.108827 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:49.110401 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:49.112556 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.112567 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.112578 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.112905 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.113209 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.113212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.113214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.113358 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.113887 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:49.113893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:49.114214 (+ 321us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.114421 (+ 207us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.114833 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":150,"thread_start_us":252,"threads_started":1}
I20260806 17:02:49.117309 16689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 17:02:49 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035769, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.124@KRBTEST.COM for oryx/127.9.111.125@KRBTEST.COM
I20260806 17:02:49.126715 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.117895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:43363 (local address 127.0.0.1:51656)
0806 17:02:49.118003 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.118019 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.118119 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.118862 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.118865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.119049 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:49.119143 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.119149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.120054 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.120060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.120617 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.120624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.121091 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.121094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.121192 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.121736 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:49.121754 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:49.123119 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:49.125290 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.125293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.125294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.125501 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.125764 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.125766 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.125767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.125894 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.126322 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:49.126324 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:49.126404 (+ 80us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.126479 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.126588 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260806 17:02:49.126873 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.118200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.9.111.125:43363)
0806 17:02:49.118505 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.118518 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.118531 (+ 13us) server_negotiation.cc:407] Connection header received
0806 17:02:49.118577 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.118579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.118618 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.118849 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.119259 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.119943 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.120744 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.120981 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.121091 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.123292 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.123295 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.123298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.123322 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.125181 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.125590 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.125593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.125594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.125682 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.125965 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.125968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.125969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.126218 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.126302 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.126560 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.126673 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":111,"threads_started":1}
I20260806 17:02:49.127797 16689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 17:02:49 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035769, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.124@KRBTEST.COM for oryx/127.9.111.124@KRBTEST.COM
I20260806 17:02:49.136845 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.128383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:46063 (local address 127.0.0.1:59194)
0806 17:02:49.128637 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.128650 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.128768 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.129000 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.129003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.129211 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:49.129279 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.129284 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.130698 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.130702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.131209 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.131216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.131785 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.131788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.131892 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.132314 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:49.132325 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:49.133752 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:49.135406 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.135411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.135413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.135599 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.135872 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.135875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.135877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.135954 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.136335 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:49.136339 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:49.136417 (+ 78us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.136520 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.136655 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":18}
I20260806 17:02:49.136845 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.128414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59194 (local address 127.9.111.124:46063)
0806 17:02:49.128595 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.128599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.128718 (+ 119us) server_negotiation.cc:407] Connection header received
0806 17:02:49.128857 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.128861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.128905 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.128989 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.129391 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.130561 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.131341 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.131675 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.131786 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.133899 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.133903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.133905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.133928 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.135280 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.135693 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.135698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.135699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.135775 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.136045 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.136048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.136050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.136216 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.136319 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.136562 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.136706 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":66}
I20260806 17:02:49.140141 16689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: Bootstrap starting.
I20260806 17:02:49.142513 16689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:49.143258 16689 log.cc:824] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:49.145203 16689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: No bootstrap required, opened a new log
I20260806 17:02:49.148417 16689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } }
I20260806 17:02:49.148628 16689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:49.148661 16689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1140d9043c1a451a99d278b0cb63d384, State: Initialized, Role: FOLLOWER
I20260806 17:02:49.149130 16689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } }
I20260806 17:02:49.149617 16695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } } }
I20260806 17:02:49.149786 16695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:02:49.150328 16689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 17:02:49.155987 16706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:02:49.156101 16706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:02:49.160120 16539 master_runner.cc:428] Detected that this master 1140d9043c1a451a99d278b0cb63d384 is joining an existing cluster
I20260806 17:02:49.160215 16539 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.111.124:46063
I20260806 17:02:49.166301 16695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:49.166523 16695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } }
I20260806 17:02:49.167569 16695 leader_election.cc:288] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aeb0d5145418420593ee76bde8104d59 (127.9.111.126:38979), b5d380aa2fee4c3b9eb65c7a0d9ebfe4 (127.9.111.125:43363)
I20260806 17:02:49.168257 15961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1140d9043c1a451a99d278b0cb63d384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb0d5145418420593ee76bde8104d59" is_pre_election: true
I20260806 17:02:49.168619 16578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1140d9043c1a451a99d278b0cb63d384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" is_pre_election: true
I20260806 17:02:49.168799 16578 raft_consensus.cc:1799] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 1 LEARNER]: Handling vote request from an unknown peer 1140d9043c1a451a99d278b0cb63d384
I20260806 17:02:49.168931 16578 raft_consensus.cc:2396] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 1140d9043c1a451a99d278b0cb63d384 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260806 17:02:49.169489 16636 leader_election.cc:302] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1140d9043c1a451a99d278b0cb63d384; no voters: aeb0d5145418420593ee76bde8104d59, b5d380aa2fee4c3b9eb65c7a0d9ebfe4
I20260806 17:02:49.170235 16695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:49.170740 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 16014
I20260806 17:02:49.171710 16695 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 17:02:49.176999 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.162876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44210 (local address 127.9.111.126:38979)
0806 17:02:49.163033 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.163039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.164194 (+ 1155us) server_negotiation.cc:407] Connection header received
0806 17:02:49.164352 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.164358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.164416 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.164492 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.165128 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.165997 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.167473 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.167799 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.167875 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.173128 (+ 5253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.173133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.173136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.173169 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.175109 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.175646 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.175650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.175652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.175744 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.176044 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.176047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.176049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.176287 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.176358 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.176694 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.176830 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 17:02:49.177588 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.162985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.0.0.1:44210)
0806 17:02:49.163977 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.163989 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.164219 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.164604 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.164607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.164833 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:49.164955 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.164961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.166305 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.166308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.167333 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.167349 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.168071 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.168075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.168171 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.169511 (+ 1340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:49.169527 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:49.170037 (+ 510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:49.175261 (+ 5224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.175264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.175267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.175532 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.175855 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.175858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.175860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.175947 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.176428 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:49.176431 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:49.176523 (+ 92us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.177244 (+ 721us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.177398 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":1,"thread_start_us":118,"threads_started":1}
I20260806 17:02:49.178704 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:35221
--local_ip_for_outbound_sockets=127.9.111.65
--tserver_master_addrs=127.9.111.126:38979
--webserver_port=39693
--webserver_interface=127.9.111.65
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.65.keytab
--principal=oryx/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.111.126:38979,127.9.111.125:43363
--tserver_master_addrs=127.9.111.126:38979,127.9.111.125:43363,127.9.111.124:46063 with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 17:02:49 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035767, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.126@KRBTEST.COM for oryx/127.9.111.124@KRBTEST.COM
I20260806 17:02:49.195204 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.163671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:43363 (local address 127.0.0.1:59504)
0806 17:02:49.164459 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.164471 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.164727 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.165038 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.165041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.165290 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:49.165423 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.165430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.166595 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.166598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.167489 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.167498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.168579 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.168583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.168691 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.190460 (+ 21769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:49.190476 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:49.190946 (+ 470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:49.193124 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.193127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.193130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.193405 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.193745 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.193748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.193751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.193861 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.194356 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:49.194360 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:49.194490 (+ 130us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.194589 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.194995 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":96,"threads_started":1}
I20260806 17:02:49.195986 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.163465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59504 (local address 127.9.111.125:43363)
0806 17:02:49.163581 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.163585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.164665 (+ 1080us) server_negotiation.cc:407] Connection header received
0806 17:02:49.164825 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.164828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.164871 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.164919 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.165597 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.166450 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.167625 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.167984 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.168057 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.191157 (+ 23100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.191161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.191164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.191195 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.192982 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.193538 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.193542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.193544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.193643 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.193970 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.193973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.193974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.194218 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.194312 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.195683 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.195830 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 17:02:49.198259 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.181874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:46063 (local address 127.0.0.1:37380)
0806 17:02:49.181998 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.182013 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.182092 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.182753 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.182756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.182972 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:49.183078 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.183084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.183922 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.183928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.184533 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.184541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.185069 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.185074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.185196 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.185718 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:49.185730 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:49.187225 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:49.196117 (+ 8892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.196120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.196123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.196427 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.196872 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.196876 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.196878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.196976 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.197432 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:49.197436 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:49.197541 (+ 105us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.197641 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.198058 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 17:02:49.198940 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.163354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37374 (local address 127.9.111.124:46063)
0806 17:02:49.163454 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.163458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.163472 (+ 14us) server_negotiation.cc:407] Connection header received
0806 17:02:49.163591 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.163597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.163658 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.163746 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.164287 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.165121 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.169865 (+ 4744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.170160 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.170234 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.192839 (+ 22605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.192845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.192851 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.192885 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.194887 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.197392 (+ 2505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.197395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.197397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.197467 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.197735 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.197738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.197740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.198147 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.198223 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.198675 (+ 452us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.198792 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 17:02:49.199543 16712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.162800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:46063 (local address 127.0.0.1:37374)
0806 17:02:49.163234 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.163249 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.163498 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.163778 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.163780 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.164029 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:49.164166 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.164174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.165233 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.165237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.169690 (+ 4453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.169700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.170414 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.170418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.170533 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.190906 (+ 20373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:49.190942 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:49.191828 (+ 886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:49.197038 (+ 5210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.197041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.197044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.197289 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.197556 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.197559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.197561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.197649 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.198283 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:49.198286 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:49.198497 (+ 211us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.199221 (+ 724us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.199367 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"spinlock_wait_cycles":1280,"thread_start_us":232,"threads_started":1}
I20260806 17:02:49.200379 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.181925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37380 (local address 127.9.111.124:46063)
0806 17:02:49.182226 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.182233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.182246 (+ 13us) server_negotiation.cc:407] Connection header received
0806 17:02:49.182307 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.182309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.182356 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.182740 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.183208 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.183804 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.184668 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.184934 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.185052 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.187401 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.187405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.187407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.187430 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.195964 (+ 8534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.196551 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.196554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.196556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.196634 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.197091 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.197093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.197096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.197320 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.198829 (+ 1509us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.200080 (+ 1251us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.200220 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260806 17:02:49.201298 15947 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.111.124:46063: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1140d9043c1a451a99d278b0cb63d384"
member_type: NON_VOTER
last_known_addr {
host: "127.9.111.124"
port: 46063
}
attrs {
promote: true
}
}
dest_uuid: "aeb0d5145418420593ee76bde8104d59"
cas_config_opid_index: 5
I20260806 17:02:49.201982 15947 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: NON_VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: true } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: NON_VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: true } }
I20260806 17:02:49.203085 16530 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1140d9043c1a451a99d278b0cb63d384 (127.9.111.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: NON_VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: true } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: NON_VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: true } } }
I20260806 17:02:49.203784 16662 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 1 FOLLOWER]: Refusing update from remote peer aeb0d5145418420593ee76bde8104d59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 17:02:49.204818 16530 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.111.124:46063
I20260806 17:02:49.204996 16530 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: NON_VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:49.205128 16527 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aeb0d5145418420593ee76bde8104d59" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: NON_VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: true } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: NON_VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: true } } }
I20260806 17:02:49.205250 16527 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:49.206025 16539 master.cc:537] Master@127.9.111.124:46063 shutting down...
W20260806 17:02:49.208668 15934 consensus_peers.cc:487] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 -> Peer 1140d9043c1a451a99d278b0cb63d384 (127.9.111.124:46063): Couldn't send request to peer 1140d9043c1a451a99d278b0cb63d384. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 17:02:49.211239 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.204646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:43363 (local address 127.0.0.1:59512)
0806 17:02:49.204893 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.204908 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.204985 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.206667 (+ 1682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.206672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.206696 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:49.206798 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.206804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.208303 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.208306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.209601 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.209613 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.210797 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.210801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.210900 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.210949 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.211026 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.211107 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260806 17:02:49.211582 16539 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:49.211710 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.205506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59512 (local address 127.9.111.125:43363)
0806 17:02:49.206290 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.206294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.206308 (+ 14us) server_negotiation.cc:407] Connection header received
0806 17:02:49.206357 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.206360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.206418 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.206509 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:49.207212 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.208163 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.209740 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.210436 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.210545 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.210727 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.211502 (+ 775us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.211587 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702}
I20260806 17:02:49.211907 16539 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:49.212024 16539 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: stopping tablet replica
I20260806 17:02:49.212136 16578 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 1 LEARNER]: Refusing update from remote peer aeb0d5145418420593ee76bde8104d59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 17:02:49.212637 16717 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: NON_VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 17:02:49.214557 16578 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1140d9043c1a451a99d278b0cb63d384 (127.9.111.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: NON_VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: true } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: NON_VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: true } } }
I20260806 17:02:49.216454 16607 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aeb0d5145418420593ee76bde8104d59. Latest consensus state: current_term: 1 leader_uuid: "aeb0d5145418420593ee76bde8104d59" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: NON_VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: true } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: NON_VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: true } } }
I20260806 17:02:49.216578 16607 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [sys.catalog]: This master's current role is: LEARNER
I20260806 17:02:49.216670 16607 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aeb0d5145418420593ee76bde8104d59" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: NON_VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: true } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: NON_VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: true } } }
I20260806 17:02:49.216764 16607 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [sys.catalog]: This master's current role is: LEARNER
I20260806 17:02:49.217643 16539 master.cc:552] Master@127.9.111.124:46063 shutdown complete.
I20260806 17:02:49.217695 16539 master_runner.cc:305] Clearing existing system tablet
I20260806 17:02:49.217983 16539 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:49.218047 16539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:49.219362 16539 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 17:02:49.220254 16721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:49.220506 16539 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:49.220646 16539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 17:02:49.220696 16539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/wal
uuid: "1140d9043c1a451a99d278b0cb63d384"
format_stamp: "Formatted at 2026-08-06 17:02:49 on dist-test-slave-69rg"
server_key: "91a30a59c68af399b78d9c9cb4d693d8"
server_key_iv: "6fecb98ab8b29a4785e052171dff2626"
server_key_version: "encryptionkey@0"
I20260806 17:02:49.222857 16539 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:49.225459 16539 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 17:02:49.225574 16539 log.cc:1196] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 17:02:49.225888 16539 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: Deleting consensus metadata
I20260806 17:02:49.226068 16539 master_runner.cc:315] Copying system tablet from 127.9.111.126:38979
I20260806 17:02:49.227010 16717 raft_consensus.cc:1050] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59: attempting to promote NON_VOTER b5d380aa2fee4c3b9eb65c7a0d9ebfe4 to VOTER
I20260806 17:02:49.227264 16539 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: tablet copy: Beginning tablet copy session from remote peer at address 127.9.111.126:38979
I20260806 17:02:49.227555 16717 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: NON_VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: true } }
I20260806 17:02:49.229342 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.229024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:46063 (local address 127.0.0.1:37382)
0806 17:02:49.229177 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.229257 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:46063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260806 17:02:49.230000 16578 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 1 LEARNER]: Refusing update from remote peer aeb0d5145418420593ee76bde8104d59: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 17:02:49.230444 16527 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 17:02:49.231689 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.231418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:46063 (local address 127.0.0.1:37394)
0806 17:02:49.231541 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.231621 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:46063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260806 17:02:49.232378 15934 consensus_peers.cc:487] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 -> Peer 1140d9043c1a451a99d278b0cb63d384 (127.9.111.124:46063): Couldn't send request to peer 1140d9043c1a451a99d278b0cb63d384. Status: Network error: Client connection negotiation failed: client connection to 127.9.111.124:46063: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 17:02:49.233403 16527 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b5d380aa2fee4c3b9eb65c7a0d9ebfe4 (127.9.111.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: NON_VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: true } } }
I20260806 17:02:49.234741 16578 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b5d380aa2fee4c3b9eb65c7a0d9ebfe4 (127.9.111.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: NON_VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: true } } }
I20260806 17:02:49.236305 16719 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aeb0d5145418420593ee76bde8104d59" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: NON_VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: true } } }
I20260806 17:02:49.236423 16719 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:02:49.236668 16527 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aeb0d5145418420593ee76bde8104d59" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: NON_VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: true } } }
I20260806 17:02:49.236783 16527 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:49.236964 16527 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aeb0d5145418420593ee76bde8104d59" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: NON_VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: true } } }
I20260806 17:02:49.237067 16527 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:49.239300 16727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.228228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.0.0.1:44226)
0806 17:02:49.228568 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.228582 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.228743 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.229098 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.229101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.229334 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:49.229453 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.229460 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.230552 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.230555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.231448 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.231462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.232308 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.232311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.232395 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.233205 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:49.233221 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:49.233674 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:49.237362 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.237366 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.237368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.237642 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.238028 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.238031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.238034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.238144 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.238625 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:49.238628 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:49.238734 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.238940 (+ 206us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.239092 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":97,"threads_started":1}
I20260806 17:02:49.239321 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.227980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44226 (local address 127.9.111.126:38979)
0806 17:02:49.228109 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.228114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.228655 (+ 541us) server_negotiation.cc:407] Connection header received
0806 17:02:49.228854 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.228857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.228904 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.228954 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.229584 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.230413 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.231587 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.231992 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.232074 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.233869 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.233873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.233875 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.233902 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.235666 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.237769 (+ 2103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.237775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.237778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.237923 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.238263 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.238267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.238268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.238505 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.238598 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.239018 (+ 420us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.239157 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 17:02:49.240096 15972 tablet_copy_service.cc:140] P aeb0d5145418420593ee76bde8104d59: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1140d9043c1a451a99d278b0cb63d384 ({username='oryx', principal='oryx/127.9.111.124@KRBTEST.COM'} at 127.0.0.1:44226)
I20260806 17:02:49.240188 15972 tablet_copy_service.cc:161] P aeb0d5145418420593ee76bde8104d59: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1140d9043c1a451a99d278b0cb63d384 at {username='oryx', principal='oryx/127.9.111.124@KRBTEST.COM'} at 127.0.0.1:44226: session id = 1140d9043c1a451a99d278b0cb63d384-00000000000000000000000000000000
I20260806 17:02:49.241214 15972 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 17:02:49.242522 16539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 17:02:49.244997 16539 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: tablet copy: Starting download of 0 data blocks...
I20260806 17:02:49.245193 16539 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: tablet copy: Starting download of 1 WAL segments...
I20260806 17:02:49.250072 16539 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 17:02:49.252315 15972 tablet_copy_service.cc:342] P aeb0d5145418420593ee76bde8104d59: Request end of tablet copy session 1140d9043c1a451a99d278b0cb63d384-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.111.124@KRBTEST.COM'} at 127.0.0.1:44226
I20260806 17:02:49.252429 15972 tablet_copy_service.cc:433] P aeb0d5145418420593ee76bde8104d59: ending tablet copy session 1140d9043c1a451a99d278b0cb63d384-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1140d9043c1a451a99d278b0cb63d384
I20260806 17:02:49.259555 16539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:49.261426 16729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:49.262990 16732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:49.263073 16539 server_base.cc:1034] running on GCE node
W20260806 17:02:49.266428 16730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:49.266765 16539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:49.267079 16286 heartbeater.cc:503] Master 127.9.111.126:38979 was elected leader, sending a full tablet report...
I20260806 17:02:49.267172 16539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:49.268317 16539 hybrid_clock.cc:648] HybridClock initialized: now 1786035769268299 us; error 37 us; skew 500 ppm
Aug 06 17:02:49 dist-test-slave-69rg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035769, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:49.269995 16539 init.cc:377] Logged in from keytab as oryx/127.9.111.124@KRBTEST.COM (short username oryx)
I20260806 17:02:49.270484 16539 webserver.cc:460] Webserver started at http://127.9.111.124:37313/ using document root <none> and password file <none>
I20260806 17:02:49.270675 16539 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:49.270718 16539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:49.272133 16539 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 17:02:49.273203 16739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:49.273440 16539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 17:02:49.273514 16539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/wal
uuid: "1140d9043c1a451a99d278b0cb63d384"
format_stamp: "Formatted at 2026-08-06 17:02:49 on dist-test-slave-69rg"
server_key: "91a30a59c68af399b78d9c9cb4d693d8"
server_key_iv: "6fecb98ab8b29a4785e052171dff2626"
server_key_version: "encryptionkey@0"
I20260806 17:02:49.273589 16539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:49.302836 16539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:49.305241 16539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:49.310706 16539 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.124:46063
I20260806 17:02:49.310743 16794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.124:46063 every 8 connection(s)
I20260806 17:02:49.311342 16539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/master-2/data/info.pb
I20260806 17:02:49.313200 16795 sys_catalog.cc:263] Verifying existing consensus state
I20260806 17:02:49.314345 16795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: Bootstrap starting.
I20260806 17:02:49.325629 16795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 17:02:49.326017 16795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: Bootstrap complete.
I20260806 17:02:49.326681 16795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: NON_VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: true } }
I20260806 17:02:49.326881 16795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1140d9043c1a451a99d278b0cb63d384, State: Initialized, Role: LEARNER
I20260806 17:02:49.326988 16795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: NON_VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: true } }
I20260806 17:02:49.327382 16798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: NON_VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: true } } }
I20260806 17:02:49.327473 16795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:49.327617 16798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [sys.catalog]: This master's current role is: LEARNER
I20260806 17:02:49.332372 16809 catalog_manager.cc:1259] Loaded cluster ID: e6a59be1451a40deb161b33abd2f6a28
I20260806 17:02:49.332460 16809 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: loading cluster ID for follower catalog manager: success
I20260806 17:02:49.335000 16809 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: acquiring CA information for follower catalog manager: success
I20260806 17:02:49.336681 16809 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 17:02:49.342448 16715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:49.342724 16715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:49.342801 16715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:49.347091 16715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:49.347160 16715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:49.347250 16715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:02:49.352838 16715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.65.keytab
--principal=oryx/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:35221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=39693
--tserver_master_addrs=127.9.111.126:38979,127.9.111.125:43363,127.9.111.124:46063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:49.354120 16715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:49.355193 16715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:49.365435 16814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:49.365513 16715 server_base.cc:1034] running on GCE node
W20260806 17:02:49.365417 16817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:49.365429 16815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:49.366170 16715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:49.366895 16715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:49.368090 16715 hybrid_clock.cc:648] HybridClock initialized: now 1786035769368074 us; error 27 us; skew 500 ppm
Aug 06 17:02:49 dist-test-slave-69rg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035769, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:49.371150 16715 init.cc:377] Logged in from keytab as oryx/127.9.111.65@KRBTEST.COM (short username oryx)
I20260806 17:02:49.372071 16715 webserver.cc:460] Webserver started at http://127.9.111.65:39693/ using document root <none> and password file <none>
I20260806 17:02:49.372701 16715 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:49.372822 16715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:49.377205 16715 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 17:02:49.379522 16824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:49.380906 16715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 17:02:49.381072 16715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "4823da74a32c4172a1f08101bc777555"
format_stamp: "Formatted at 2026-08-06 17:02:47 on dist-test-slave-69rg"
server_key: "3dab996652b814ffaba2a82db43e3e49"
server_key_iv: "7385949edabea958ceb99927e3c8369a"
server_key_version: "encryptionkey@0"
I20260806 17:02:49.381742 16715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:49.403956 16715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:49.407647 16715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:49.407871 16715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:49.408495 16715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:49.409439 16715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:49.409511 16715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:49.409579 16715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:49.409631 16715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:49.424088 16715 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:35221
I20260806 17:02:49.424198 16942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:35221 every 8 connection(s)
I20260806 17:02:49.425209 16715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:02:49.432312 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 16715
I20260806 17:02:49.432547 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 16152
Aug 06 17:02:49 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035769, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.125@KRBTEST.COM
Aug 06 17:02:49 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035769, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM
Aug 06 17:02:49 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035769, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.124@KRBTEST.COM
I20260806 17:02:49.444764 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.427403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:56137 (local address 127.9.111.125:43363)
0806 17:02:49.427550 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.427555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.428792 (+ 1237us) server_negotiation.cc:407] Connection header received
0806 17:02:49.430530 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.430533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.430588 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.430695 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.432352 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.433140 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.435139 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.435488 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.435563 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.440020 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.440025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.440027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.440056 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.442099 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.442792 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.442796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.442798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.442894 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.443333 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.443336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.443338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.443591 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.443675 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.444409 (+ 734us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.444552 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 17:02:49.444788 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:34495
--local_ip_for_outbound_sockets=127.9.111.66
--tserver_master_addrs=127.9.111.126:38979
--webserver_port=42435
--webserver_interface=127.9.111.66
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.66.keytab
--principal=oryx/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.111.126:38979,127.9.111.125:43363
--tserver_master_addrs=127.9.111.126:38979,127.9.111.125:43363,127.9.111.124:46063 with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:02:49.445019 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.429527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:59459 (local address 127.9.111.126:38979)
0806 17:02:49.429664 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.429669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.429731 (+ 62us) server_negotiation.cc:407] Connection header received
0806 17:02:49.430500 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.430504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.430564 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.430665 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.432319 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.433143 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.435139 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.435488 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.435563 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.440127 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.440134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.440140 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.440178 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.442929 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.443494 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.443501 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.443505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.443609 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.443946 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.443952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.443957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.444215 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.444306 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.444685 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.444822 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 17:02:49.446187 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.427974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:44963 (local address 127.9.111.124:46063)
0806 17:02:49.428277 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.428282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.428792 (+ 510us) server_negotiation.cc:407] Connection header received
0806 17:02:49.430530 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.430534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.430588 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.430780 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.436458 (+ 5678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.437496 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.438777 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.439247 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.439386 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.441791 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.441795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.441798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.441866 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.443850 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.444317 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.444322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.444326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.444418 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.444713 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.444716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.444717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.445331 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.445420 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.445802 (+ 382us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.445976 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":133,"threads_started":1}
I20260806 17:02:49.446658 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.427862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:43363 (local address 127.9.111.65:56137)
0806 17:02:49.428499 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.428561 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.430271 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.430915 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.430920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.431475 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:49.432187 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.432199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.433341 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.433345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.434997 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.435007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.435763 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.435767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.436039 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.437206 (+ 1167us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:49.437221 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:49.439497 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:49.442225 (+ 2728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.442231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.442256 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.442663 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.443007 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.443013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.443016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.443211 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.443751 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:49.443759 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:49.444124 (+ 365us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.444962 (+ 838us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.445389 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":332,"spinlock_wait_cycles":7168,"thread_start_us":133,"threads_started":1}
I20260806 17:02:49.446942 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.427862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:46063 (local address 127.9.111.65:44963)
0806 17:02:49.428499 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.428561 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.430271 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.430973 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.430976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.431475 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:49.436263 (+ 4788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.436280 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.437634 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.437640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.438618 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.438632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.439419 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.439425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.439517 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.440018 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:49.440033 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:49.441624 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:49.443960 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.443962 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.443964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.444187 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.444530 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.444534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.444536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.444616 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.445475 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:49.445478 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:49.445564 (+ 86us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.446437 (+ 873us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.446577 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":95,"spinlock_wait_cycles":30976,"thread_start_us":278,"threads_started":1}
I20260806 17:02:49.447391 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.429046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.65:59459)
0806 17:02:49.429471 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.429487 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.430275 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.430866 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.430873 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.431474 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:49.432110 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.432123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.433341 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.433345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.434998 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.435007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.435763 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.435767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.436040 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.436888 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:49.436919 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:49.439945 (+ 3026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:49.443078 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.443084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.443089 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.443365 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.443742 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.443747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.443752 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.443844 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.444384 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:49.444387 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:49.444477 (+ 90us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.445432 (+ 955us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.445640 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":353,"spinlock_wait_cycles":4480,"thread_start_us":185,"threads_started":1}
I20260806 17:02:49.448454 16943 heartbeater.cc:347] Connected to a master server at 127.9.111.124:46063
I20260806 17:02:49.448493 16944 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38979
I20260806 17:02:49.448707 16943 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:49.448709 16944 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:49.448444 16946 heartbeater.cc:347] Connected to a master server at 127.9.111.125:43363
I20260806 17:02:49.449362 16944 heartbeater.cc:511] Master 127.9.111.126:38979 requested a full tablet report, sending...
I20260806 17:02:49.449359 16943 heartbeater.cc:511] Master 127.9.111.124:46063 requested a full tablet report, sending...
I20260806 17:02:49.449455 16946 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:49.449609 16946 heartbeater.cc:511] Master 127.9.111.125:43363 requested a full tablet report, sending...
I20260806 17:02:49.450946 16568 ts_manager.cc:194] Registered new tserver with Master: 4823da74a32c4172a1f08101bc777555 (127.9.111.65:35221)
I20260806 17:02:49.451017 16757 ts_manager.cc:194] Registered new tserver with Master: 4823da74a32c4172a1f08101bc777555 (127.9.111.65:35221)
I20260806 17:02:49.451730 15947 ts_manager.cc:194] Re-registered known tserver with Master: 4823da74a32c4172a1f08101bc777555 (127.9.111.65:35221)
I20260806 17:02:49.452378 15947 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:59459
I20260806 17:02:49.514873 16424 heartbeater.cc:503] Master 127.9.111.126:38979 was elected leader, sending a full tablet report...
W20260806 17:02:49.566136 16952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:49.566416 16952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:49.566500 16952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:49.570709 16952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:49.570780 16952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:49.570842 16952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:02:49.576507 16952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.66.keytab
--principal=oryx/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:34495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=42435
--tserver_master_addrs=127.9.111.126:38979,127.9.111.125:43363,127.9.111.124:46063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:49.577697 16952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:49.578881 16952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:49.589118 16957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:49.589190 16960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:49.589219 16958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:49.589730 16952 server_base.cc:1034] running on GCE node
I20260806 17:02:49.590199 16952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:49.590866 16952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:49.592089 16952 hybrid_clock.cc:648] HybridClock initialized: now 1786035769592052 us; error 47 us; skew 500 ppm
Aug 06 17:02:49 dist-test-slave-69rg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035769, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:49.595486 16952 init.cc:377] Logged in from keytab as oryx/127.9.111.66@KRBTEST.COM (short username oryx)
I20260806 17:02:49.596585 16952 webserver.cc:460] Webserver started at http://127.9.111.66:42435/ using document root <none> and password file <none>
I20260806 17:02:49.597208 16952 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:49.597340 16952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:49.602016 16952 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:49.604318 16967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:49.605928 16952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:49.606096 16952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "fe05064e06e544d48bd43c56811ceca1"
format_stamp: "Formatted at 2026-08-06 17:02:48 on dist-test-slave-69rg"
server_key: "281757a5931364491d816a1ae388284c"
server_key_iv: "07100954a941df9292d12c0142bded75"
server_key_version: "encryptionkey@0"
I20260806 17:02:49.606724 16952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:49.625627 16952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:49.629809 16952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:49.630096 16952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:49.630864 16952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:49.631886 16952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:49.631964 16952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:49.632048 16952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:49.632107 16952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:49.643342 16952 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:34495
I20260806 17:02:49.643391 17081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:34495 every 8 connection(s)
I20260806 17:02:49.644510 16952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:02:49.645412 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 16952
I20260806 17:02:49.645586 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 16292
I20260806 17:02:49.653944 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:33877
--local_ip_for_outbound_sockets=127.9.111.67
--tserver_master_addrs=127.9.111.126:38979
--webserver_port=42349
--webserver_interface=127.9.111.67
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.67.keytab
--principal=oryx/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.111.126:38979,127.9.111.125:43363
--tserver_master_addrs=127.9.111.126:38979,127.9.111.125:43363,127.9.111.124:46063 with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 17:02:49 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035769, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.125@KRBTEST.COM
Aug 06 17:02:49 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035769, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.124@KRBTEST.COM
Aug 06 17:02:49 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035769, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM
I20260806 17:02:49.668598 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.646814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:60435 (local address 127.9.111.126:38979)
0806 17:02:49.649503 (+ 2689us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.649555 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.649711 (+ 156us) server_negotiation.cc:407] Connection header received
0806 17:02:49.651091 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.651095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.651160 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.651226 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.653618 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.654566 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.656939 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.657332 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.657424 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.663229 (+ 5805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.663236 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.663241 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.663285 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.665349 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.666304 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.666308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.666310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.666425 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.667013 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.667017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.667019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.667284 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.667379 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.668253 (+ 874us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.668405 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2556}
I20260806 17:02:49.668598 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.649319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:34925 (local address 127.9.111.124:46063)
0806 17:02:49.649869 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.649876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.650033 (+ 157us) server_negotiation.cc:407] Connection header received
0806 17:02:49.651091 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.651095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.651171 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.651228 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.653618 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.654566 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.656939 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.657348 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.657426 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.662478 (+ 5052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.662484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.662491 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.662528 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.665378 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.666303 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.666308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.666310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.666543 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.667013 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.667017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.667019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.667284 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.667379 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.668253 (+ 874us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.668406 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425}
I20260806 17:02:49.669178 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.649548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:45103 (local address 127.9.111.125:43363)
0806 17:02:49.649698 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.649705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.650179 (+ 474us) server_negotiation.cc:407] Connection header received
0806 17:02:49.651083 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.651090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.651164 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.651230 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.653602 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.654474 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.656956 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.657385 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.657491 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.663647 (+ 6156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.663655 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.663660 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.663702 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.665864 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.667297 (+ 1433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.667315 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.667320 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.667450 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.667899 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.667905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.667910 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.668208 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.668314 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.668781 (+ 467us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.668935 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 17:02:49.670583 17087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.647984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.66:60435)
0806 17:02:49.649340 (+ 1356us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.649408 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.650809 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.651715 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.651718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.652442 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:49.653404 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.653428 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.654758 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.654763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.656765 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.656779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.657664 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.657668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.658029 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.659124 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:49.659140 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:49.662915 (+ 3775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:49.665519 (+ 2604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.665527 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.665550 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.666163 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.666711 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.666767 (+ 56us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.666769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.666885 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.667470 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:49.667472 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:49.667893 (+ 421us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.668881 (+ 988us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.669325 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1094,"mutex_wait_us":593,"spinlock_wait_cycles":46976,"thread_start_us":160,"threads_started":1}
I20260806 17:02:49.670590 17088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.649136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:43363 (local address 127.9.111.66:45103)
0806 17:02:49.650077 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.650095 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.650835 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.651688 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.651700 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.652471 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:49.653405 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.653428 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.654693 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.654700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.656789 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.656802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.657693 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.657699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.658035 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.658927 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:49.658956 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:49.663431 (+ 4475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:49.666022 (+ 2591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.666029 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.666035 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.667143 (+ 1108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.667643 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.667649 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.667653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.667766 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.668408 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:49.668411 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:49.668540 (+ 129us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.669427 (+ 887us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.669722 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":697,"spinlock_wait_cycles":20096,"thread_start_us":81,"threads_started":1}
I20260806 17:02:49.670591 17086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.648844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:46063 (local address 127.9.111.66:34925)
0806 17:02:49.649771 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.649786 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.650815 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.651688 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.651698 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.652452 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:49.653404 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.653427 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.655018 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.655022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.656792 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.656799 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.657664 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.657668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.658035 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.658992 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:49.659011 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:49.662243 (+ 3232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:49.665519 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.665527 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.665550 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.666163 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.666656 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.666663 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.666669 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.666885 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.667461 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:49.667468 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:49.667893 (+ 425us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.669452 (+ 1559us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.669601 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":466,"spinlock_wait_cycles":90368,"thread_start_us":130,"threads_started":1}
I20260806 17:02:49.672178 17083 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38979
I20260806 17:02:49.672271 17084 heartbeater.cc:347] Connected to a master server at 127.9.111.125:43363
I20260806 17:02:49.672441 17083 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:49.672978 17083 heartbeater.cc:511] Master 127.9.111.126:38979 requested a full tablet report, sending...
I20260806 17:02:49.673617 17082 heartbeater.cc:347] Connected to a master server at 127.9.111.124:46063
I20260806 17:02:49.673712 17082 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:49.673969 17082 heartbeater.cc:511] Master 127.9.111.124:46063 requested a full tablet report, sending...
I20260806 17:02:49.672441 17084 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:49.674314 15947 ts_manager.cc:194] Re-registered known tserver with Master: fe05064e06e544d48bd43c56811ceca1 (127.9.111.66:34495)
I20260806 17:02:49.674372 17084 heartbeater.cc:511] Master 127.9.111.125:43363 requested a full tablet report, sending...
I20260806 17:02:49.674783 16757 ts_manager.cc:194] Registered new tserver with Master: fe05064e06e544d48bd43c56811ceca1 (127.9.111.66:34495)
I20260806 17:02:49.675050 16568 ts_manager.cc:194] Registered new tserver with Master: fe05064e06e544d48bd43c56811ceca1 (127.9.111.66:34495)
I20260806 17:02:49.675154 15947 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:60435
W20260806 17:02:49.787236 17089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:49.787567 17089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:49.787653 17089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:49.792028 17089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:49.792135 17089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:49.792235 17089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:02:49.797972 17089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.67.keytab
--principal=oryx/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:33877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=42349
--tserver_master_addrs=127.9.111.126:38979,127.9.111.125:43363,127.9.111.124:46063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:49.799244 17089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:49.800313 17089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:49.810439 17096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:49.810563 17089 server_base.cc:1034] running on GCE node
W20260806 17:02:49.810510 17099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:49.810510 17097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:49.811149 17089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:49.811772 17089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:49.812965 17089 hybrid_clock.cc:648] HybridClock initialized: now 1786035769812953 us; error 27 us; skew 500 ppm
Aug 06 17:02:49 dist-test-slave-69rg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035769, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:49.815907 17089 init.cc:377] Logged in from keytab as oryx/127.9.111.67@KRBTEST.COM (short username oryx)
I20260806 17:02:49.816932 17089 webserver.cc:460] Webserver started at http://127.9.111.67:42349/ using document root <none> and password file <none>
I20260806 17:02:49.817561 17089 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:49.817643 17089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:49.821552 17089 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:49.823933 17107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:49.825306 17089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 17:02:49.825429 17089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "c84fd741df5e4071835c818d2e769a0d"
format_stamp: "Formatted at 2026-08-06 17:02:48 on dist-test-slave-69rg"
server_key: "311289b67cff36e881196945ceb230b3"
server_key_iv: "4e930eeaa68cb2ebffd77bddffd3b591"
server_key_version: "encryptionkey@0"
I20260806 17:02:49.826105 17089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:49.836721 17089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:49.840116 17089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:49.840301 17089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:49.840857 17089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:49.841696 17089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:49.841742 17089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:49.841782 17089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:49.841797 17089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:49.854142 17089 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:33877
I20260806 17:02:49.854167 17222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:33877 every 8 connection(s)
I20260806 17:02:49.854830 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.850663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:46063 (local address 127.0.0.1:37400)
0806 17:02:49.850925 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.850942 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.851184 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.851636 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.851642 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.851667 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:49.851795 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.851801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.852565 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.852569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.853565 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.853576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.854389 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.854393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.854484 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.854508 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.854594 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.854678 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":81,"threads_started":1}
I20260806 17:02:49.854825 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.850718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37400 (local address 127.9.111.124:46063)
0806 17:02:49.850923 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.850928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.851151 (+ 223us) server_negotiation.cc:407] Connection header received
0806 17:02:49.851311 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.851315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.851421 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.851616 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:49.851912 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.852447 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.853687 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.854208 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.854344 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.854507 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.854608 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.854678 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 17:02:49.855418 17089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:02:49.855481 16767 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 17:02:49.856088 17223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aeb0d5145418420593ee76bde8104d59. Latest consensus state: current_term: 1 leader_uuid: "aeb0d5145418420593ee76bde8104d59" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: NON_VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: true } } }
I20260806 17:02:49.856503 17223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [sys.catalog]: This master's current role is: LEARNER
I20260806 17:02:49.863328 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 17089
Aug 06 17:02:49 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035769, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM
Aug 06 17:02:49 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035769, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.125@KRBTEST.COM
Aug 06 17:02:49 dist-test-slave-69rg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035769, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.124@KRBTEST.COM
I20260806 17:02:49.876178 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.857705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:50167 (local address 127.9.111.124:46063)
0806 17:02:49.857845 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.857849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.859049 (+ 1200us) server_negotiation.cc:407] Connection header received
0806 17:02:49.860137 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.860140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.860198 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.860295 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.861930 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.862459 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.864423 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.864777 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.864846 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.870656 (+ 5810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.870660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.870663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.870691 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.873571 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.874294 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.874298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.874299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.874389 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.874828 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.874832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.874836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.875087 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.875173 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.875869 (+ 696us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.876004 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":49}
I20260806 17:02:49.877427 17230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.859404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:43363 (local address 127.9.111.67:52365)
0806 17:02:49.860351 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.860366 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.860572 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.860821 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.860823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.861192 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:49.861748 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.861762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.862884 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.862887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.864284 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.864292 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.865018 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.865022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.865345 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.866283 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:49.866304 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:49.869981 (+ 3677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:49.874580 (+ 4599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.874583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.874584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.874819 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.875109 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.875115 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.875121 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.875212 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.875650 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:49.875653 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:49.875729 (+ 76us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.875845 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.876195 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":278,"spinlock_wait_cycles":4864,"thread_start_us":86,"threads_started":1}
I20260806 17:02:49.877667 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.858195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:46063 (local address 127.9.111.67:50167)
0806 17:02:49.858767 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.858821 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.859935 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.860475 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.860482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.861192 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:49.861749 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.861762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.862598 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.862601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.864281 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.864292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.865018 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.865022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.865349 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.866442 (+ 1093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:49.866459 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:49.870500 (+ 4041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:49.873713 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.873738 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.873759 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.874164 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.874495 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.874501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.874506 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.874696 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.875225 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:49.875232 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:49.875567 (+ 335us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.876320 (+ 753us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.876448 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":364,"spinlock_wait_cycles":13952,"thread_start_us":248,"threads_started":1}
I20260806 17:02:49.878280 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.859594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:52365 (local address 127.9.111.125:43363)
0806 17:02:49.859748 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.859752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.860582 (+ 830us) server_negotiation.cc:407] Connection header received
0806 17:02:49.860641 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.860647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.860701 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.860799 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.861962 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.862761 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.864423 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.864768 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.864846 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.870164 (+ 5318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.870168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.870173 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.870219 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.874471 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.874924 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.874927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.874929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.875006 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.875326 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.875329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.875331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.875520 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.875608 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.877902 (+ 2294us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.878129 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 17:02:49.878638 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.866186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44240 (local address 127.9.111.126:38979)
0806 17:02:49.866976 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.866981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.868229 (+ 1248us) server_negotiation.cc:407] Connection header received
0806 17:02:49.868283 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.868286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.868340 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.868542 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.869319 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.870002 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.871191 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.871597 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.871697 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.873878 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.873882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.873885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.873911 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.876214 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.876798 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.876804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.876809 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.876919 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.877193 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.877197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.877199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.877389 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.877535 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.878334 (+ 799us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.878480 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":66,"threads_started":1}
I20260806 17:02:49.879024 17224 heartbeater.cc:347] Connected to a master server at 127.9.111.124:46063
I20260806 17:02:49.879110 17226 heartbeater.cc:347] Connected to a master server at 127.9.111.125:43363
I20260806 17:02:49.879268 17224 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:49.879268 17226 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:49.879920 17224 heartbeater.cc:511] Master 127.9.111.124:46063 requested a full tablet report, sending...
I20260806 17:02:49.879920 17226 heartbeater.cc:511] Master 127.9.111.125:43363 requested a full tablet report, sending...
I20260806 17:02:49.880584 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.858195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.67:41541)
0806 17:02:49.858766 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.858821 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.859952 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.860916 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.860919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.861183 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:49.861747 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.861760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.862912 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.862915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.864234 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.864246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.865019 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.865025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.865349 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.866343 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:49.866363 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:49.870153 (+ 3790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:49.878609 (+ 8456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.878612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.878614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.878879 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.879206 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:49.879209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:49.879212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:49.879322 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:49.879856 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:49.879861 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:49.879969 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.880067 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.880267 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":204,"spinlock_wait_cycles":40192,"thread_start_us":207,"threads_started":1}
I20260806 17:02:49.881088 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.857699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:41541 (local address 127.9.111.126:38979)
0806 17:02:49.857936 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.857942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.859049 (+ 1107us) server_negotiation.cc:407] Connection header received
0806 17:02:49.860145 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.860149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.860209 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.860290 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.861956 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.862768 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.864394 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.864719 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.864804 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.870295 (+ 5491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.870298 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.870301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.870330 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.873652 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.879000 (+ 5348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.879003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.879005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.879095 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.879446 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.879453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.879457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.879701 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.879806 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.880793 (+ 987us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.880911 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260806 17:02:49.881124 16568 ts_manager.cc:194] Registered new tserver with Master: c84fd741df5e4071835c818d2e769a0d (127.9.111.67:33877)
I20260806 17:02:49.881124 16757 ts_manager.cc:194] Registered new tserver with Master: c84fd741df5e4071835c818d2e769a0d (127.9.111.67:33877)
I20260806 17:02:49.881810 17225 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38979
I20260806 17:02:49.881918 17225 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:49.882078 17225 heartbeater.cc:511] Master 127.9.111.126:38979 requested a full tablet report, sending...
I20260806 17:02:49.882774 15947 ts_manager.cc:194] Re-registered known tserver with Master: c84fd741df5e4071835c818d2e769a0d (127.9.111.67:33877)
I20260806 17:02:49.883248 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.865799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37408 (local address 127.9.111.124:46063)
0806 17:02:49.866143 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.866149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.871000 (+ 4851us) server_negotiation.cc:407] Connection header received
0806 17:02:49.871189 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.871192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.871238 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.871443 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.872397 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.873155 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.874430 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.874878 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.879071 (+ 4193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.879190 (+ 119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.879194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.879196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.879214 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.881153 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.881650 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.881653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.881655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.881764 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.882150 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.882156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.882159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.882396 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.882534 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.882968 (+ 434us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.883091 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1509,"server-negotiator.queue_time_us":259,"thread_start_us":108,"threads_started":1}
I20260806 17:02:49.883525 15947 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:41541
I20260806 17:02:49.887552 17239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.867139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.9.111.125:43363)
0806 17:02:49.867536 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.867540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.868079 (+ 539us) server_negotiation.cc:407] Connection header received
0806 17:02:49.868292 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.868295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.868348 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.868587 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:49.869606 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.870794 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.872229 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.872551 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.872644 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.883254 (+ 10610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:49.883258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:49.883262 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:49.883290 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:49.885088 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.885574 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.885579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.885581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.886170 (+ 589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:49.886613 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:49.886617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:49.886619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:49.886845 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:49.886982 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.887247 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.887376 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":128,"threads_started":1}
I20260806 17:02:49.886775 15947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:02:49.890301 15947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:02:49.893013 17090 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aeb0d5145418420593ee76bde8104d59" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: NON_VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: true } } }
I20260806 17:02:49.893181 17090 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:49.896730 17242 raft_consensus.cc:1050] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59: attempting to promote NON_VOTER 1140d9043c1a451a99d278b0cb63d384 to VOTER
I20260806 17:02:49.897110 17242 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: false } }
I20260806 17:02:49.898391 16578 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 1 FOLLOWER]: Refusing update from remote peer aeb0d5145418420593ee76bde8104d59: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 17:02:49.898731 17250 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 17:02:49.899845 16767 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 1 LEARNER]: Refusing update from remote peer aeb0d5145418420593ee76bde8104d59: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 17:02:49.900225 17250 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 17:02:49.901060 17250 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1140d9043c1a451a99d278b0cb63d384 (127.9.111.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: false } } }
I20260806 17:02:49.901604 16578 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1140d9043c1a451a99d278b0cb63d384 (127.9.111.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: false } } }
I20260806 17:02:49.903043 16767 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1140d9043c1a451a99d278b0cb63d384 (127.9.111.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: false } } }
I20260806 17:02:49.903352 17090 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aeb0d5145418420593ee76bde8104d59" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: false } } }
I20260806 17:02:49.903457 17090 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:49.903558 17090 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aeb0d5145418420593ee76bde8104d59" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: false } } }
I20260806 17:02:49.903663 17090 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:49.905306 17223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aeb0d5145418420593ee76bde8104d59" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: false } } }
I20260806 17:02:49.905431 17223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:02:49.906092 17252 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aeb0d5145418420593ee76bde8104d59" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: false } } }
I20260806 17:02:49.906225 17252 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:02:49.907655 17090 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aeb0d5145418420593ee76bde8104d59" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: false } } }
I20260806 17:02:49.907769 17090 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:49.912192 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.906291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:34495 (local address 127.0.0.1:55086)
0806 17:02:49.906615 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.906632 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.906713 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.907967 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.907970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.907989 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:49.908112 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.908119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.909453 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.909456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.910632 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.910642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.911762 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.911765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.911859 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.911881 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.911958 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.912045 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":56}
I20260806 17:02:49.912407 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.906925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:33877 (local address 127.0.0.1:45772)
0806 17:02:49.907978 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.907990 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.908064 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.908711 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.908714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.908732 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:49.908834 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.908841 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.909924 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.909928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.911043 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.911054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.912024 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.912028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.912103 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.912124 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.912182 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.912269 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"mutex_wait_us":48,"thread_start_us":106,"threads_started":1}
I20260806 17:02:49.912837 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.907110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.9.111.67:33877)
0806 17:02:49.908314 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.908324 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.908342 (+ 18us) server_negotiation.cc:407] Connection header received
0806 17:02:49.908400 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.908403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.908580 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.908732 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:49.908927 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.909780 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.911180 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.911921 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.912027 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.912178 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.912230 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.912669 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1096,"thread_start_us":120,"threads_started":1}
I20260806 17:02:49.912956 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.906635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55086 (local address 127.9.111.66:34495)
0806 17:02:49.907452 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.907463 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.907483 (+ 20us) server_negotiation.cc:407] Connection header received
0806 17:02:49.907578 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.907585 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.907826 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.908007 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:49.908214 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.909334 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.910783 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.911612 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.912476 (+ 864us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.912636 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.912691 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.912809 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":118,"threads_started":1}
I20260806 17:02:49.913224 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.906681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:35221 (local address 127.0.0.1:57412)
0806 17:02:49.907366 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:49.907380 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:49.907472 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:49.908814 (+ 1342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:49.908817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:49.908837 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:49.908928 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.908934 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.910363 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.910366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.911678 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:49.911690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.912821 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.912825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:49.912914 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.912937 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 17:02:49.913018 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 17:02:49.913098 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
I20260806 17:02:49.913275 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:49.907074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57412 (local address 127.9.111.65:35221)
0806 17:02:49.908287 (+ 1213us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:49.908302 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:49.908324 (+ 22us) server_negotiation.cc:407] Connection header received
0806 17:02:49.908393 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:49.908397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:49.908590 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:49.909063 (+ 473us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:49.909180 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.910266 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.911809 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:49.912697 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:49.912813 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:49.912966 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:49.913048 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 17:02:49.913143 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1104,"thread_start_us":69,"threads_started":1}
I20260806 17:02:49.915993 17016 tablet_service.cc:1467] Processing CreateTablet for tablet 3203caaf1146428cb4f967834f4d5e92 (DEFAULT_TABLE table=test-table [id=b0fa43352f7d4f12826d0c956b190c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:49.916024 16874 tablet_service.cc:1467] Processing CreateTablet for tablet 3203caaf1146428cb4f967834f4d5e92 (DEFAULT_TABLE table=test-table [id=b0fa43352f7d4f12826d0c956b190c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:49.917164 17016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3203caaf1146428cb4f967834f4d5e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:49.917179 16874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3203caaf1146428cb4f967834f4d5e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:49.924528 17259 tablet_bootstrap.cc:492] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1: Bootstrap starting.
I20260806 17:02:49.916026 17155 tablet_service.cc:1467] Processing CreateTablet for tablet 3203caaf1146428cb4f967834f4d5e92 (DEFAULT_TABLE table=test-table [id=b0fa43352f7d4f12826d0c956b190c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:49.926007 17260 tablet_bootstrap.cc:492] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555: Bootstrap starting.
I20260806 17:02:49.926400 17155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3203caaf1146428cb4f967834f4d5e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:49.927531 17259 tablet_bootstrap.cc:654] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:49.928336 17260 tablet_bootstrap.cc:654] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:49.928547 17259 log.cc:824] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:49.929440 17260 log.cc:824] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:49.930950 17259 tablet_bootstrap.cc:492] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1: No bootstrap required, opened a new log
I20260806 17:02:49.931228 17259 ts_tablet_manager.cc:1397] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 17:02:49.935359 17263 tablet_bootstrap.cc:492] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d: Bootstrap starting.
I20260806 17:02:49.936734 17259 raft_consensus.cc:348] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } } peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } }
I20260806 17:02:49.936990 17259 raft_consensus.cc:374] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:49.937047 17259 raft_consensus.cc:729] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe05064e06e544d48bd43c56811ceca1, State: Initialized, Role: FOLLOWER
I20260806 17:02:49.937507 17259 consensus_queue.cc:260] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } } peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } }
I20260806 17:02:49.938192 17263 tablet_bootstrap.cc:654] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:49.938511 17259 ts_tablet_manager.cc:1428] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260806 17:02:49.938282 17260 tablet_bootstrap.cc:492] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555: No bootstrap required, opened a new log
I20260806 17:02:49.938764 17260 ts_tablet_manager.cc:1397] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260806 17:02:49.939291 17083 heartbeater.cc:503] Master 127.9.111.126:38979 was elected leader, sending a full tablet report...
I20260806 17:02:49.939477 17263 log.cc:824] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:49.941906 17263 tablet_bootstrap.cc:492] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d: No bootstrap required, opened a new log
I20260806 17:02:49.942109 17263 ts_tablet_manager.cc:1397] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 17:02:49.945802 17260 raft_consensus.cc:348] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } } peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } }
I20260806 17:02:49.946130 17260 raft_consensus.cc:374] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:49.946188 17260 raft_consensus.cc:729] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4823da74a32c4172a1f08101bc777555, State: Initialized, Role: FOLLOWER
I20260806 17:02:49.946625 17260 consensus_queue.cc:260] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } } peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } }
I20260806 17:02:49.946585 17263 raft_consensus.cc:348] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } } peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } }
I20260806 17:02:49.946759 17263 raft_consensus.cc:374] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:49.946843 17263 raft_consensus.cc:729] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84fd741df5e4071835c818d2e769a0d, State: Initialized, Role: FOLLOWER
I20260806 17:02:49.947320 16944 heartbeater.cc:503] Master 127.9.111.126:38979 was elected leader, sending a full tablet report...
I20260806 17:02:49.947310 17263 consensus_queue.cc:260] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } } peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } }
I20260806 17:02:49.948338 17263 ts_tablet_manager.cc:1428] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 17:02:49.949241 17260 ts_tablet_manager.cc:1428] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555: Time spent starting tablet: real 0.010s user 0.004s sys 0.003s
I20260806 17:02:49.949455 17225 heartbeater.cc:503] Master 127.9.111.126:38979 was elected leader, sending a full tablet report...
W20260806 17:02:50.107645 17227 tablet.cc:2369] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 17:02:50.146721 17085 tablet.cc:2369] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 17:02:50.177981 16947 tablet.cc:2369] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:02:50.270547 17264 raft_consensus.cc:482] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:50.270804 17264 raft_consensus.cc:504] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } } peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } }
I20260806 17:02:50.272055 17264 leader_election.cc:288] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4823da74a32c4172a1f08101bc777555 (127.9.111.65:35221), c84fd741df5e4071835c818d2e769a0d (127.9.111.67:33877)
I20260806 17:02:50.277297 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:50.272477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:57473 (local address 127.9.111.67:33877)
0806 17:02:50.272667 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:50.272672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:50.272812 (+ 140us) server_negotiation.cc:407] Connection header received
0806 17:02:50.273287 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:50.273290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:50.273366 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:50.273466 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:50.273715 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:50.274476 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:50.275884 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:50.276498 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:50.276625 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:50.276678 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:50.277077 (+ 399us) server_negotiation.cc:299] Negotiation successful
0806 17:02:50.277165 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260806 17:02:50.277295 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:50.272554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:42499 (local address 127.9.111.65:35221)
0806 17:02:50.272667 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:50.272672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:50.272771 (+ 99us) server_negotiation.cc:407] Connection header received
0806 17:02:50.272966 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:50.272975 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:50.273020 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:50.273135 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:50.273384 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:50.274241 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:50.275884 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:50.276498 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:50.276625 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:50.276678 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:50.277077 (+ 399us) server_negotiation.cc:299] Negotiation successful
0806 17:02:50.277169 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 17:02:50.277320 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:50.272349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:35221 (local address 127.9.111.66:42499)
0806 17:02:50.272662 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:50.272679 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:50.272813 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:50.273154 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:50.273156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:50.273173 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:50.273265 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:50.273270 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:50.274355 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:50.274358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:50.275768 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:50.275776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:50.276628 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:50.276631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:50.276867 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:50.276926 (+ 59us) client_negotiation.cc:769] Sending connection context
0806 17:02:50.277015 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 17:02:50.277118 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":136,"spinlock_wait_cycles":13952,"thread_start_us":80,"threads_started":1}
I20260806 17:02:50.277354 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:50.272356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:33877 (local address 127.9.111.66:57473)
0806 17:02:50.272663 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:50.272679 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:50.272813 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:50.273501 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:50.273504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:50.273523 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:50.273611 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:50.273615 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:50.274572 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:50.274575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:50.275740 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:50.275753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:50.276628 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:50.276631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:50.276930 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:50.276948 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 17:02:50.277015 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 17:02:50.277118 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":86,"spinlock_wait_cycles":2176,"thread_start_us":85,"threads_started":1}
I20260806 17:02:50.278182 16895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3203caaf1146428cb4f967834f4d5e92" candidate_uuid: "fe05064e06e544d48bd43c56811ceca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4823da74a32c4172a1f08101bc777555" is_pre_election: true
I20260806 17:02:50.278326 17176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3203caaf1146428cb4f967834f4d5e92" candidate_uuid: "fe05064e06e544d48bd43c56811ceca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84fd741df5e4071835c818d2e769a0d" is_pre_election: true
I20260806 17:02:50.278558 16895 raft_consensus.cc:2454] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe05064e06e544d48bd43c56811ceca1 in term 0.
I20260806 17:02:50.278678 17176 raft_consensus.cc:2454] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe05064e06e544d48bd43c56811ceca1 in term 0.
I20260806 17:02:50.279106 16969 leader_election.cc:302] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4823da74a32c4172a1f08101bc777555, fe05064e06e544d48bd43c56811ceca1; no voters:
I20260806 17:02:50.279412 17264 raft_consensus.cc:2781] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:02:50.279520 17264 raft_consensus.cc:482] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:02:50.279584 17264 raft_consensus.cc:3037] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:50.280795 17264 raft_consensus.cc:504] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } } peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } }
I20260806 17:02:50.281213 17264 leader_election.cc:288] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [CANDIDATE]: Term 1 election: Requested vote from peers 4823da74a32c4172a1f08101bc777555 (127.9.111.65:35221), c84fd741df5e4071835c818d2e769a0d (127.9.111.67:33877)
I20260806 17:02:50.281625 16895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3203caaf1146428cb4f967834f4d5e92" candidate_uuid: "fe05064e06e544d48bd43c56811ceca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4823da74a32c4172a1f08101bc777555"
I20260806 17:02:50.281744 16895 raft_consensus.cc:3037] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:50.281775 17176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3203caaf1146428cb4f967834f4d5e92" candidate_uuid: "fe05064e06e544d48bd43c56811ceca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84fd741df5e4071835c818d2e769a0d"
I20260806 17:02:50.281944 17176 raft_consensus.cc:3037] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:50.283074 16895 raft_consensus.cc:2454] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe05064e06e544d48bd43c56811ceca1 in term 1.
I20260806 17:02:50.283087 17176 raft_consensus.cc:2454] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe05064e06e544d48bd43c56811ceca1 in term 1.
I20260806 17:02:50.283375 16969 leader_election.cc:302] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4823da74a32c4172a1f08101bc777555, fe05064e06e544d48bd43c56811ceca1; no voters:
I20260806 17:02:50.283574 17264 raft_consensus.cc:2781] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:50.283802 17264 raft_consensus.cc:686] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [term 1 LEADER]: Becoming Leader. State: Replica: fe05064e06e544d48bd43c56811ceca1, State: Running, Role: LEADER
I20260806 17:02:50.284178 17264 consensus_queue.cc:237] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } } peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } }
I20260806 17:02:50.288607 15945 catalog_manager.cc:5495] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe05064e06e544d48bd43c56811ceca1 (127.9.111.66). New cstate: current_term: 1 leader_uuid: "fe05064e06e544d48bd43c56811ceca1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } health_report { overall_health: UNKNOWN } } }
I20260806 17:02:50.363989 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:50.360049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55100 (local address 127.9.111.66:34495)
0806 17:02:50.360229 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:50.360233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:50.360248 (+ 15us) server_negotiation.cc:407] Connection header received
0806 17:02:50.360347 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:50.360350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:50.360409 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:50.360509 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:50.360801 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:50.361365 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:50.362272 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:50.362487 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:50.362569 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:50.362921 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:50.363546 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:50.363653 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:50.363769 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 17:02:50.363836 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 17:02:50.370317 16895 raft_consensus.cc:1261] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555 [term 1 FOLLOWER]: Refusing update from remote peer fe05064e06e544d48bd43c56811ceca1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:50.370335 17176 raft_consensus.cc:1261] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d [term 1 FOLLOWER]: Refusing update from remote peer fe05064e06e544d48bd43c56811ceca1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:50.371150 17271 consensus_queue.cc:1035] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:50.371364 17264 consensus_queue.cc:1035] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:50.381561 17275 mvcc.cc:205] Tried to move back new op lower bound from 7315602515428466688 to 7315602515086839808. Current Snapshot: MvccSnapshot[applied={T|T < 7315602515428466688}]
I20260806 17:02:50.391124 15949 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:50.391350 15949 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:50.395951 15949 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aeb0d5145418420593ee76bde8104d59: Sending DeleteTablet for 3 replicas of tablet 3203caaf1146428cb4f967834f4d5e92
I20260806 17:02:50.396884 16874 tablet_service.cc:1514] Processing DeleteTablet for tablet 3203caaf1146428cb4f967834f4d5e92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:50 UTC) from {username='oryx', principal='oryx/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:57412
I20260806 17:02:50.397550 17016 tablet_service.cc:1514] Processing DeleteTablet for tablet 3203caaf1146428cb4f967834f4d5e92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:50 UTC) from {username='oryx', principal='oryx/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:55086
I20260806 17:02:50.398062 17286 tablet_replica.cc:330] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1: stopping tablet replica
I20260806 17:02:50.398420 17286 raft_consensus.cc:2229] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:02:50.398599 17155 tablet_service.cc:1514] Processing DeleteTablet for tablet 3203caaf1146428cb4f967834f4d5e92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:50 UTC) from {username='oryx', principal='oryx/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:45772
I20260806 17:02:50.398780 17286 raft_consensus.cc:2258] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:50.400411 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 15916
I20260806 17:02:50.401695 17285 tablet_replica.cc:330] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555: stopping tablet replica
I20260806 17:02:50.402127 17285 raft_consensus.cc:2229] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:50.402513 17285 raft_consensus.cc:2258] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:50.402776 17287 tablet_replica.cc:330] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d: stopping tablet replica
I20260806 17:02:50.403079 17287 raft_consensus.cc:2229] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:50.403384 17287 raft_consensus.cc:2258] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:50.405110 17287 ts_tablet_manager.cc:1905] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:50.405120 17285 ts_tablet_manager.cc:1905] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:50.408161 17287 ts_tablet_manager.cc:1918] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:02:50.408372 17287 log.cc:1196] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/3203caaf1146428cb4f967834f4d5e92
I20260806 17:02:50.408793 17287 ts_tablet_manager.cc:1939] T 3203caaf1146428cb4f967834f4d5e92 P c84fd741df5e4071835c818d2e769a0d: Deleting consensus metadata
I20260806 17:02:50.408810 17285 ts_tablet_manager.cc:1918] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:02:50.408916 17285 log.cc:1196] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-0/wal/wals/3203caaf1146428cb4f967834f4d5e92
I20260806 17:02:50.409263 17285 ts_tablet_manager.cc:1939] T 3203caaf1146428cb4f967834f4d5e92 P 4823da74a32c4172a1f08101bc777555: Deleting consensus metadata
I20260806 17:02:50.410926 17286 ts_tablet_manager.cc:1905] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:50.413348 17286 ts_tablet_manager.cc:1918] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:02:50.413439 17286 log.cc:1196] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/3203caaf1146428cb4f967834f4d5e92
I20260806 17:02:50.413748 17286 ts_tablet_manager.cc:1939] T 3203caaf1146428cb4f967834f4d5e92 P fe05064e06e544d48bd43c56811ceca1: Deleting consensus metadata
W20260806 17:02:50.415033 17111 connection.cc:535] server connection from 127.0.0.1:45772 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 17:02:50.415066 16827 connection.cc:535] server connection from 127.0.0.1:57412 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 17:02:50.416013 16970 connection.cc:535] server connection from 127.0.0.1:55086 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 17:02:50.432392 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:50.399838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.9.111.125:43363)
0806 17:02:50.400082 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:50.400103 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:50.416264 (+ 16161us) server_negotiation.cc:407] Connection header received
0806 17:02:50.416389 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:50.416392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:50.416463 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:50.416567 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:50.417490 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:50.418292 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:50.419436 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:50.419783 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:50.419871 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:50.427683 (+ 7812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:50.427687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:50.427689 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:50.427717 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:50.430183 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:50.430863 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:50.430867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:50.430868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:50.430961 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:50.431334 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:50.431338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:50.431340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:50.431561 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:50.431667 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:50.432077 (+ 410us) server_negotiation.cc:299] Negotiation successful
0806 17:02:50.432218 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":75,"threads_started":1}
I20260806 17:02:50.433101 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:50.419121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59534 (local address 127.9.111.125:43363)
0806 17:02:50.422059 (+ 2938us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:50.422062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:50.422269 (+ 207us) server_negotiation.cc:407] Connection header received
0806 17:02:50.422410 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:50.422414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:50.422467 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:50.422550 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:50.423275 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:50.424027 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:50.425161 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:50.425522 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:50.425617 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:50.427605 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:50.427609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:50.427612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:50.427639 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:50.430060 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:50.430730 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:50.430734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:50.430736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:50.430824 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:50.431160 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:50.431164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:50.431165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:50.431515 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:50.431641 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:50.432356 (+ 715us) server_negotiation.cc:299] Negotiation successful
0806 17:02:50.432939 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2865,"spinlock_wait_cycles":896,"thread_start_us":61,"threads_started":1}
I20260806 17:02:50.438966 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:50.419110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.9.111.124:46063)
0806 17:02:50.419348 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:50.419355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:50.419866 (+ 511us) server_negotiation.cc:407] Connection header received
0806 17:02:50.419915 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:50.419918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:50.419974 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:50.420065 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:50.420931 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:50.421457 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:50.422699 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:50.423059 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:50.423162 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:50.435112 (+ 11950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:50.435116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:50.435119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:50.435149 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:50.437026 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:50.437533 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:50.437536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:50.437538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:50.437629 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:50.437976 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:50.437980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:50.437982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:50.438209 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:50.438321 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:50.438666 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 17:02:50.438797 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":64,"threads_started":1}
I20260806 17:02:50.444147 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:50.399966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.9.111.124:46063)
0806 17:02:50.400320 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:50.400324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:50.433876 (+ 33552us) server_negotiation.cc:407] Connection header received
0806 17:02:50.434207 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:50.434211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:50.434271 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:50.434378 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:50.435072 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:50.436219 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:50.437358 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:50.437703 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:50.437767 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:50.439618 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:50.439622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:50.439626 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:50.439657 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:50.441337 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:50.441787 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:50.441791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:50.441792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:50.441905 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:50.442398 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:50.442401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:50.442402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:50.442592 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:50.443049 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:50.443867 (+ 818us) server_negotiation.cc:299] Negotiation successful
0806 17:02:50.443985 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":79,"threads_started":1}
I20260806 17:02:51.386251 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.385541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.65:46123)
0806 17:02:51.386021 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.386132 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":128,"threads_started":1}
W20260806 17:02:51.386807 16944 heartbeater.cc:644] Failed to heartbeat to 127.9.111.126:38979 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
I20260806 17:02:51.392102 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.391681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.67:45067)
0806 17:02:51.391919 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.392002 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":67,"threads_started":1}
W20260806 17:02:51.392473 17225 heartbeater.cc:644] Failed to heartbeat to 127.9.111.126:38979 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
I20260806 17:02:51.392619 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.392176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.66:33691)
0806 17:02:51.392451 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.392517 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":79,"threads_started":1}
W20260806 17:02:51.392863 17083 heartbeater.cc:644] Failed to heartbeat to 127.9.111.126:38979 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
I20260806 17:02:51.896152 17304 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aeb0d5145418420593ee76bde8104d59)
I20260806 17:02:51.896373 17304 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: false } }
I20260806 17:02:51.897373 17304 leader_election.cc:288] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aeb0d5145418420593ee76bde8104d59 (127.9.111.126:38979), 1140d9043c1a451a99d278b0cb63d384 (127.9.111.124:46063)
I20260806 17:02:51.897519 17305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aeb0d5145418420593ee76bde8104d59)
I20260806 17:02:51.897632 17305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: false } }
I20260806 17:02:51.898227 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.897695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.0.0.1:44732)
0806 17:02:51.898006 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.898087 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":43,"threads_started":1}
W20260806 17:02:51.898469 16553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
I20260806 17:02:51.898775 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.898256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.0.0.1:44740)
0806 17:02:51.898567 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.898648 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":67,"threads_started":1}
W20260806 17:02:51.898948 16742 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
I20260806 17:02:51.899245 17305 leader_election.cc:288] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aeb0d5145418420593ee76bde8104d59 (127.9.111.126:38979), b5d380aa2fee4c3b9eb65c7a0d9ebfe4 (127.9.111.125:43363)
I20260806 17:02:51.899923 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.899641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.0.0.1:44752)
0806 17:02:51.899777 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.899829 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260806 17:02:51.900094 16553 leader_election.cc:334] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aeb0d5145418420593ee76bde8104d59 (127.9.111.126:38979): Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
I20260806 17:02:51.900842 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.899667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.0.0.1:44758)
0806 17:02:51.899799 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.899853 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260806 17:02:51.900941 16742 leader_election.cc:334] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aeb0d5145418420593ee76bde8104d59 (127.9.111.126:38979): Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
I20260806 17:02:51.902611 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.897559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:46063 (local address 127.0.0.1:37442)
0806 17:02:51.897964 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.897980 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:51.898091 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:51.898543 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:51.898549 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:51.898573 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:51.898680 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:51.898688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.899741 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.899746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:51.901340 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:51.901350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.902261 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.902264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:51.902335 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:51.902356 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 17:02:51.902442 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 17:02:51.902498 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
I20260806 17:02:51.902655 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.897761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37442 (local address 127.9.111.124:46063)
0806 17:02:51.898193 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:51.898199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:51.898213 (+ 14us) server_negotiation.cc:407] Connection header received
0806 17:02:51.898264 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:51.898267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:51.898321 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:51.898413 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:51.898828 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.899614 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.901462 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.902082 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.902191 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:51.902249 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:51.902473 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 17:02:51.902546 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":130,"threads_started":1}
I20260806 17:02:51.903138 16767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1140d9043c1a451a99d278b0cb63d384" is_pre_election: true
I20260806 17:02:51.903349 16767 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5d380aa2fee4c3b9eb65c7a0d9ebfe4 in term 1.
I20260806 17:02:51.903753 16554 leader_election.cc:302] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1140d9043c1a451a99d278b0cb63d384, b5d380aa2fee4c3b9eb65c7a0d9ebfe4; no voters: aeb0d5145418420593ee76bde8104d59
I20260806 17:02:51.903990 17304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 17:02:51.904058 17304 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aeb0d5145418420593ee76bde8104d59)
I20260806 17:02:51.904100 17304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 17:02:51.904181 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.898666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:43363 (local address 127.0.0.1:59540)
0806 17:02:51.899082 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.899095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:51.899183 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:51.899673 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:51.899676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:51.899695 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:51.899783 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:51.899791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.901638 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.901641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:51.902812 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:51.902821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.903783 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.903787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:51.903879 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:51.903902 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 17:02:51.903984 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 17:02:51.904051 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":71,"threads_started":1}
I20260806 17:02:51.904232 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.898870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59540 (local address 127.9.111.125:43363)
0806 17:02:51.899272 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:51.899285 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:51.899355 (+ 70us) server_negotiation.cc:407] Connection header received
0806 17:02:51.899400 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:51.899403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:51.899442 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:51.899542 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:51.899921 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.901546 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.902932 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.903539 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.903655 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:51.903723 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:51.904016 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 17:02:51.904112 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":321,"thread_start_us":96,"threads_started":1}
I20260806 17:02:51.904601 16578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1140d9043c1a451a99d278b0cb63d384" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" is_pre_election: true
I20260806 17:02:51.905383 17304 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: false } }
I20260806 17:02:51.905606 16578 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1140d9043c1a451a99d278b0cb63d384 in current term 2: Already voted for candidate b5d380aa2fee4c3b9eb65c7a0d9ebfe4 in this term.
I20260806 17:02:51.905799 17304 leader_election.cc:288] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [CANDIDATE]: Term 2 election: Requested vote from peers aeb0d5145418420593ee76bde8104d59 (127.9.111.126:38979), 1140d9043c1a451a99d278b0cb63d384 (127.9.111.124:46063)
I20260806 17:02:51.905994 16742 leader_election.cc:302] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1140d9043c1a451a99d278b0cb63d384; no voters: aeb0d5145418420593ee76bde8104d59, b5d380aa2fee4c3b9eb65c7a0d9ebfe4
I20260806 17:02:51.906186 16767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1140d9043c1a451a99d278b0cb63d384"
I20260806 17:02:51.906307 16767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 1 FOLLOWER]: Advancing to term 2
I20260806 17:02:51.906366 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.905951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.0.0.1:44768)
0806 17:02:51.906206 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.906302 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260806 17:02:51.906999 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.906791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.0.0.1:44770)
0806 17:02:51.906897 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.906943 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 17:02:51.907111 16553 leader_election.cc:334] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aeb0d5145418420593ee76bde8104d59 (127.9.111.126:38979): Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
I20260806 17:02:51.907552 16767 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5d380aa2fee4c3b9eb65c7a0d9ebfe4 in term 2.
I20260806 17:02:51.907678 17305 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 17:02:51.907922 16554 leader_election.cc:302] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1140d9043c1a451a99d278b0cb63d384, b5d380aa2fee4c3b9eb65c7a0d9ebfe4; no voters: aeb0d5145418420593ee76bde8104d59
I20260806 17:02:51.908092 17304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 17:02:51.908372 17304 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [term 2 LEADER]: Becoming Leader. State: Replica: b5d380aa2fee4c3b9eb65c7a0d9ebfe4, State: Running, Role: LEADER
I20260806 17:02:51.908635 17304 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: false } }
I20260806 17:02:51.910125 17314 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5d380aa2fee4c3b9eb65c7a0d9ebfe4. Latest consensus state: current_term: 2 leader_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: false } } }
I20260806 17:02:51.910231 17314 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:51.910542 17316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:51.912931 17316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:02:51.913316 17316 catalog_manager.cc:1259] Loaded cluster ID: e6a59be1451a40deb161b33abd2f6a28
I20260806 17:02:51.913360 17316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:51.913977 17316 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:51.914372 17316 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: Loaded TSK: 0
I20260806 17:02:51.915030 17316 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:02:51.960647 16567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:02:51.963850 16567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:02:51.966915 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.966649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.0.0.1:44790)
0806 17:02:51.966772 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.966835 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 17:02:51.967025 16767 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [term 2 FOLLOWER]: Refusing update from remote peer b5d380aa2fee4c3b9eb65c7a0d9ebfe4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260806 17:02:51.967618 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.967356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.0.0.1:44792)
0806 17:02:51.967461 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.967514 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 17:02:51.967532 17304 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260806 17:02:51.967859 16553 consensus_peers.cc:487] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 -> Peer aeb0d5145418420593ee76bde8104d59 (127.9.111.126:38979): Couldn't send request to peer aeb0d5145418420593ee76bde8104d59. Status: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 17:02:51.969111 17305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5d380aa2fee4c3b9eb65c7a0d9ebfe4. Latest consensus state: current_term: 2 leader_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: false } } }
I20260806 17:02:51.969231 17305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:02:51.970300 17304 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: false } } }
I20260806 17:02:51.970443 17304 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:51.970599 17305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aeb0d5145418420593ee76bde8104d59" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38979 } } peers { permanent_uuid: "b5d380aa2fee4c3b9eb65c7a0d9ebfe4" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 43363 } attrs { promote: false } } peers { permanent_uuid: "1140d9043c1a451a99d278b0cb63d384" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 46063 } attrs { promote: false } } }
I20260806 17:02:51.970705 17305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1140d9043c1a451a99d278b0cb63d384 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:02:51.971542 17318 mvcc.cc:205] Tried to move back new op lower bound from 7315602521971154944 to 7315602521740238848. Current Snapshot: MvccSnapshot[applied={T|T < 7315602521971154944}]
I20260806 17:02:51.980713 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.976589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:33877 (local address 127.0.0.1:45780)
0806 17:02:51.976732 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.976743 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:51.976836 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:51.977261 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:51.977264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:51.977281 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:51.977412 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:51.977419 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.978539 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.978544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:51.979445 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:51.979455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.980315 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.980319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:51.980409 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:51.980434 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 17:02:51.980506 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 17:02:51.980575 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 17:02:51.981124 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.976701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45780 (local address 127.9.111.67:33877)
0806 17:02:51.976975 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:51.976980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:51.976995 (+ 15us) server_negotiation.cc:407] Connection header received
0806 17:02:51.977050 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:51.977053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:51.977130 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:51.977532 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:51.977668 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.978426 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.979566 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.980187 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.980295 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:51.980360 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:51.980898 (+ 538us) server_negotiation.cc:299] Negotiation successful
0806 17:02:51.980982 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":93,"threads_started":1}
I20260806 17:02:51.981551 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.977150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:35221 (local address 127.0.0.1:57428)
0806 17:02:51.977231 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.977245 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:51.977328 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:51.978038 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:51.978042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:51.978087 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:51.978190 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:51.978196 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.979104 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.979107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:51.980060 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:51.980067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.981215 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.981219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:51.981312 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:51.981331 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 17:02:51.981373 (+ 42us) client_negotiation.cc:240] Negotiation successful
0806 17:02:51.981423 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260806 17:02:51.981729 17155 tablet_service.cc:1467] Processing CreateTablet for tablet 33de450d875e4529916532a6e6088e1f (DEFAULT_TABLE table=test-table [id=4fc48fad178e436a9a1a50d3641bf129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:51.982060 17155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33de450d875e4529916532a6e6088e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:51.982064 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.977497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57428 (local address 127.9.111.65:35221)
0806 17:02:51.977754 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:51.977758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:51.977773 (+ 15us) server_negotiation.cc:407] Connection header received
0806 17:02:51.977851 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:51.977855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:51.977914 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:51.978308 (+ 394us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:51.978419 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.978996 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.980345 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.981080 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.981717 (+ 637us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:51.981785 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:51.981853 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 17:02:51.981929 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":69,"threads_started":1}
I20260806 17:02:51.982527 16874 tablet_service.cc:1467] Processing CreateTablet for tablet 33de450d875e4529916532a6e6088e1f (DEFAULT_TABLE table=test-table [id=4fc48fad178e436a9a1a50d3641bf129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:51.982786 16874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33de450d875e4529916532a6e6088e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:51.983559 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.977150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:34495 (local address 127.0.0.1:55114)
0806 17:02:51.977710 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.977723 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:51.977800 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:51.979577 (+ 1777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:51.979580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:51.979597 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:51.979667 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:51.979671 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.980784 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.980788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:51.981962 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:51.981972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.983175 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.983178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:51.983262 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:51.983283 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 17:02:51.983351 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 17:02:51.983412 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":289,"thread_start_us":106,"threads_started":1}
I20260806 17:02:51.983937 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.977283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55114 (local address 127.9.111.66:34495)
0806 17:02:51.978301 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:51.978306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:51.978320 (+ 14us) server_negotiation.cc:407] Connection header received
0806 17:02:51.978373 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:51.978376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:51.978425 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:51.978530 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:51.979791 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.980650 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.982092 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:51.982911 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:51.983019 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:51.983110 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:51.983718 (+ 608us) server_negotiation.cc:299] Negotiation successful
0806 17:02:51.983803 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":93,"threads_started":1}
I20260806 17:02:51.984457 17327 tablet_bootstrap.cc:492] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d: Bootstrap starting.
I20260806 17:02:51.985371 17328 tablet_bootstrap.cc:492] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555: Bootstrap starting.
I20260806 17:02:51.985416 17327 tablet_bootstrap.cc:654] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:51.985765 17016 tablet_service.cc:1467] Processing CreateTablet for tablet 33de450d875e4529916532a6e6088e1f (DEFAULT_TABLE table=test-table [id=4fc48fad178e436a9a1a50d3641bf129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:51.986076 17016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33de450d875e4529916532a6e6088e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:51.986270 17328 tablet_bootstrap.cc:654] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:51.987407 17328 tablet_bootstrap.cc:492] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555: No bootstrap required, opened a new log
I20260806 17:02:51.987516 17328 ts_tablet_manager.cc:1397] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 17:02:51.988004 17328 raft_consensus.cc:348] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } } peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } }
I20260806 17:02:51.988129 17328 raft_consensus.cc:374] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:51.988169 17328 raft_consensus.cc:729] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4823da74a32c4172a1f08101bc777555, State: Initialized, Role: FOLLOWER
I20260806 17:02:51.988291 17328 consensus_queue.cc:260] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } } peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } }
I20260806 17:02:51.988655 17331 tablet_bootstrap.cc:492] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1: Bootstrap starting.
I20260806 17:02:51.988690 17328 ts_tablet_manager.cc:1428] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 17:02:51.989544 17327 tablet_bootstrap.cc:492] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d: No bootstrap required, opened a new log
I20260806 17:02:51.989638 17327 ts_tablet_manager.cc:1397] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 17:02:51.990067 17327 raft_consensus.cc:348] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } } peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } }
I20260806 17:02:51.990219 17327 raft_consensus.cc:374] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:51.990257 17327 raft_consensus.cc:729] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84fd741df5e4071835c818d2e769a0d, State: Initialized, Role: FOLLOWER
I20260806 17:02:51.990383 17327 consensus_queue.cc:260] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } } peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } }
I20260806 17:02:51.990710 17331 tablet_bootstrap.cc:654] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:51.990721 17327 ts_tablet_manager.cc:1428] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 17:02:51.991402 16946 heartbeater.cc:503] Master 127.9.111.125:43363 was elected leader, sending a full tablet report...
I20260806 17:02:51.991815 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.991359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.67:45211)
0806 17:02:51.991656 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.991718 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":100,"threads_started":1}
I20260806 17:02:51.992919 17226 heartbeater.cc:503] Master 127.9.111.125:43363 was elected leader, sending a full tablet report...
I20260806 17:02:51.993321 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.993097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.67:52647)
0806 17:02:51.993198 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.993252 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260806 17:02:51.993664 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.989090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.65:55361)
0806 17:02:51.989362 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.989424 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":108,"threads_started":1}
I20260806 17:02:51.994524 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.994106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.65:53183)
0806 17:02:51.994407 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.994453 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":121,"threads_started":1}
I20260806 17:02:51.994632 17331 tablet_bootstrap.cc:492] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1: No bootstrap required, opened a new log
I20260806 17:02:51.994730 17331 ts_tablet_manager.cc:1397] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260806 17:02:51.995082 17331 raft_consensus.cc:348] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } } peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } }
I20260806 17:02:51.995172 17331 raft_consensus.cc:374] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:51.995201 17331 raft_consensus.cc:729] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe05064e06e544d48bd43c56811ceca1, State: Initialized, Role: FOLLOWER
I20260806 17:02:51.995296 17331 consensus_queue.cc:260] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } } peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } }
I20260806 17:02:51.995572 17331 ts_tablet_manager.cc:1428] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 17:02:51.996208 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.995818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.66:38867)
0806 17:02:51.996062 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.996122 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":97,"threads_started":1}
I20260806 17:02:51.996870 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:51.996639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.66:58429)
0806 17:02:51.996736 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:51.996791 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 17:02:51.997148 17084 heartbeater.cc:503] Master 127.9.111.125:43363 was elected leader, sending a full tablet report...
I20260806 17:02:52.034278 17332 raft_consensus.cc:482] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:52.034432 17332 raft_consensus.cc:504] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } } peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } }
I20260806 17:02:52.035681 17332 leader_election.cc:288] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84fd741df5e4071835c818d2e769a0d (127.9.111.67:33877), fe05064e06e544d48bd43c56811ceca1 (127.9.111.66:34495)
I20260806 17:02:52.040036 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:52.036111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:32865 (local address 127.9.111.67:33877)
0806 17:02:52.036238 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:52.036243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:52.036261 (+ 18us) server_negotiation.cc:407] Connection header received
0806 17:02:52.036318 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:52.036322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:52.036376 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:52.036453 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:52.036823 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:52.037438 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:52.038926 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:52.039321 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:52.039415 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:52.039503 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:52.039810 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 17:02:52.039895 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 17:02:52.040036 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:52.035993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:34495 (local address 127.9.111.65:54579)
0806 17:02:52.036196 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:52.036210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:52.036312 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:52.036597 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:52.036599 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:52.036615 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:52.036704 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:52.036709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:52.037655 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:52.037658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:52.038703 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:52.038714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:52.039516 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:52.039523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:52.039698 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:52.039717 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 17:02:52.039829 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 17:02:52.039898 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":157}
I20260806 17:02:52.040048 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:52.035879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:33877 (local address 127.9.111.65:32865)
0806 17:02:52.036040 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:52.036058 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:52.036192 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:52.036495 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:52.036502 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:52.036527 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:52.036687 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:52.036696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:52.037553 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:52.037558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:52.038765 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:52.038802 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:52.039483 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:52.039487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:52.039612 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:52.039684 (+ 72us) client_negotiation.cc:769] Sending connection context
0806 17:02:52.039753 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 17:02:52.039858 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":76}
I20260806 17:02:52.040118 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:52.036119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:54579 (local address 127.9.111.66:34495)
0806 17:02:52.036281 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:52.036288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:52.036308 (+ 20us) server_negotiation.cc:407] Connection header received
0806 17:02:52.036398 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:52.036400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:52.036476 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:52.036559 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:52.036822 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:52.037539 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:52.038855 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:52.039346 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:52.039503 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:52.039561 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:52.039874 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 17:02:52.039967 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 17:02:52.040745 17036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33de450d875e4529916532a6e6088e1f" candidate_uuid: "4823da74a32c4172a1f08101bc777555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe05064e06e544d48bd43c56811ceca1" is_pre_election: true
I20260806 17:02:52.040913 17176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33de450d875e4529916532a6e6088e1f" candidate_uuid: "4823da74a32c4172a1f08101bc777555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84fd741df5e4071835c818d2e769a0d" is_pre_election: true
I20260806 17:02:52.041029 17036 raft_consensus.cc:2454] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4823da74a32c4172a1f08101bc777555 in term 0.
I20260806 17:02:52.041108 17176 raft_consensus.cc:2454] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4823da74a32c4172a1f08101bc777555 in term 0.
I20260806 17:02:52.041522 16827 leader_election.cc:302] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4823da74a32c4172a1f08101bc777555, c84fd741df5e4071835c818d2e769a0d, fe05064e06e544d48bd43c56811ceca1; no voters:
I20260806 17:02:52.041910 17332 raft_consensus.cc:2781] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:02:52.041970 17332 raft_consensus.cc:482] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:02:52.041997 17332 raft_consensus.cc:3037] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:52.043067 17332 raft_consensus.cc:504] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } } peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } }
I20260806 17:02:52.043458 17332 leader_election.cc:288] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [CANDIDATE]: Term 1 election: Requested vote from peers c84fd741df5e4071835c818d2e769a0d (127.9.111.67:33877), fe05064e06e544d48bd43c56811ceca1 (127.9.111.66:34495)
I20260806 17:02:52.043913 17036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33de450d875e4529916532a6e6088e1f" candidate_uuid: "4823da74a32c4172a1f08101bc777555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe05064e06e544d48bd43c56811ceca1"
I20260806 17:02:52.043999 17176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33de450d875e4529916532a6e6088e1f" candidate_uuid: "4823da74a32c4172a1f08101bc777555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84fd741df5e4071835c818d2e769a0d"
I20260806 17:02:52.044046 17036 raft_consensus.cc:3037] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:52.044152 17176 raft_consensus.cc:3037] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:52.045091 17176 raft_consensus.cc:2454] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4823da74a32c4172a1f08101bc777555 in term 1.
I20260806 17:02:52.045425 17036 raft_consensus.cc:2454] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4823da74a32c4172a1f08101bc777555 in term 1.
I20260806 17:02:52.045428 16827 leader_election.cc:302] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4823da74a32c4172a1f08101bc777555, c84fd741df5e4071835c818d2e769a0d; no voters:
I20260806 17:02:52.045738 17332 raft_consensus.cc:2781] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:52.046022 17332 raft_consensus.cc:686] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [term 1 LEADER]: Becoming Leader. State: Replica: 4823da74a32c4172a1f08101bc777555, State: Running, Role: LEADER
I20260806 17:02:52.046315 17332 consensus_queue.cc:237] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } } peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } }
I20260806 17:02:52.046873 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:52.046585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.65:46533)
0806 17:02:52.046721 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:52.046787 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 17:02:52.048614 16567 catalog_manager.cc:5495] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4823da74a32c4172a1f08101bc777555 (127.9.111.65). New cstate: current_term: 1 leader_uuid: "4823da74a32c4172a1f08101bc777555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4823da74a32c4172a1f08101bc777555" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 35221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260806 17:02:52.062731 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:52.057897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57440 (local address 127.9.111.65:35221)
0806 17:02:52.058029 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:52.058032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:52.058047 (+ 15us) server_negotiation.cc:407] Connection header received
0806 17:02:52.058159 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:52.058162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:52.058216 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:52.058311 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:52.058522 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:52.059090 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:52.060539 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:52.060991 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:52.061120 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:52.061466 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:52.062213 (+ 747us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:52.062371 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:52.062480 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 17:02:52.062570 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 17:02:52.066520 17036 raft_consensus.cc:1261] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1 [term 1 FOLLOWER]: Refusing update from remote peer 4823da74a32c4172a1f08101bc777555: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:52.066763 17176 raft_consensus.cc:1261] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d [term 1 FOLLOWER]: Refusing update from remote peer 4823da74a32c4172a1f08101bc777555: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:52.067239 17332 consensus_queue.cc:1035] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe05064e06e544d48bd43c56811ceca1" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:52.067456 17340 consensus_queue.cc:1035] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84fd741df5e4071835c818d2e769a0d" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:52.070601 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:52.069795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.66:60663)
0806 17:02:52.070451 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:52.070515 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578}
I20260806 17:02:52.070953 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:52.070276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.67:60205)
0806 17:02:52.070843 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:52.070895 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485}
I20260806 17:02:52.073663 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:52.072749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.65:52625)
0806 17:02:52.073533 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:52.073586 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":710}
I20260806 17:02:52.073838 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:52.073398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.67:46445)
0806 17:02:52.073687 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:52.073753 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222}
I20260806 17:02:52.075086 17344 mvcc.cc:205] Tried to move back new op lower bound from 7315602522376720384 to 7315602522304258048. Current Snapshot: MvccSnapshot[applied={T|T < 7315602522376720384}]
I20260806 17:02:52.075728 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:52.075445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38979 (local address 127.9.111.66:38447)
0806 17:02:52.075594 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:52.075662 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:38979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 17:02:52.075846 17345 mvcc.cc:205] Tried to move back new op lower bound from 7315602522376720384 to 7315602522304258048. Current Snapshot: MvccSnapshot[applied={T|T < 7315602522376720384 or (T in {7315602522376720384})}]
I20260806 17:02:52.081805 16568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:52.082039 16568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:52.086050 16568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5d380aa2fee4c3b9eb65c7a0d9ebfe4: Sending DeleteTablet for 3 replicas of tablet 33de450d875e4529916532a6e6088e1f
I20260806 17:02:52.086874 17016 tablet_service.cc:1514] Processing DeleteTablet for tablet 33de450d875e4529916532a6e6088e1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:52 UTC) from {username='oryx', principal='oryx/127.9.111.125@KRBTEST.COM'} at 127.0.0.1:55114
I20260806 17:02:52.086877 16874 tablet_service.cc:1514] Processing DeleteTablet for tablet 33de450d875e4529916532a6e6088e1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:52 UTC) from {username='oryx', principal='oryx/127.9.111.125@KRBTEST.COM'} at 127.0.0.1:57428
I20260806 17:02:52.086923 17155 tablet_service.cc:1514] Processing DeleteTablet for tablet 33de450d875e4529916532a6e6088e1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:52 UTC) from {username='oryx', principal='oryx/127.9.111.125@KRBTEST.COM'} at 127.0.0.1:45780
I20260806 17:02:52.087330 17353 tablet_replica.cc:330] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555: stopping tablet replica
I20260806 17:02:52.087460 17353 raft_consensus.cc:2229] T 33de450d875e4529916532a6e6088e1f P 4823da74a32c4172a1f08101bc777555 [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:02:52.087718 17354 tablet_replica.cc:330] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1: stopping tablet replica
I20260806 17:02:52.087850 17354 raft_consensus.cc:2229] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:52.087990 17354 raft_consensus.cc:2258] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:52.088197 17355 tablet_replica.cc:330] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d: stopping tablet replica
I20260806 17:02:52.088341 17355 raft_consensus.cc:2229] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:52.088387 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 16715
I20260806 17:02:52.088480 17355 raft_consensus.cc:2258] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:52.089453 17355 ts_tablet_manager.cc:1905] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:52.090668 17354 ts_tablet_manager.cc:1905] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:52.092690 17354 ts_tablet_manager.cc:1918] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:02:52.092804 17354 log.cc:1196] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/33de450d875e4529916532a6e6088e1f
I20260806 17:02:52.092877 17355 ts_tablet_manager.cc:1918] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:02:52.092969 17355 log.cc:1196] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/33de450d875e4529916532a6e6088e1f
I20260806 17:02:52.093150 17354 ts_tablet_manager.cc:1939] T 33de450d875e4529916532a6e6088e1f P fe05064e06e544d48bd43c56811ceca1: Deleting consensus metadata
I20260806 17:02:52.093258 17355 ts_tablet_manager.cc:1939] T 33de450d875e4529916532a6e6088e1f P c84fd741df5e4071835c818d2e769a0d: Deleting consensus metadata
I20260806 17:02:52.093909 16552 catalog_manager.cc:4841] TS fe05064e06e544d48bd43c56811ceca1 (127.9.111.66:34495): tablet 33de450d875e4529916532a6e6088e1f (table test-table [id=4fc48fad178e436a9a1a50d3641bf129]) successfully deleted
I20260806 17:02:52.094053 16554 catalog_manager.cc:4841] TS c84fd741df5e4071835c818d2e769a0d (127.9.111.67:33877): tablet 33de450d875e4529916532a6e6088e1f (table test-table [id=4fc48fad178e436a9a1a50d3641bf129]) successfully deleted
I20260806 17:02:52.097417 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 16952
I20260806 17:02:52.097872 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:52.097495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:35221 (local address 127.0.0.1:57442)
0806 17:02:52.097687 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:52.097753 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.65:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260806 17:02:52.098188 16553 catalog_manager.cc:4584] TS 4823da74a32c4172a1f08101bc777555 (127.9.111.65:35221): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33de450d875e4529916532a6e6088e1f: Network error: Client connection negotiation failed: client connection to 127.9.111.65:35221: connect: Connection refused (error 111)
I20260806 17:02:52.106108 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 17089
I20260806 17:02:52.113303 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 16429
I20260806 17:02:52.120582 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 16539
2026-08-06T17:02:52Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5837 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:02:52 dist-test-slave-69rg krb5kdc[17358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:02:52 dist-test-slave-69rg krb5kdc[17358](info): set up 2 sockets
Aug 06 17:02:52 dist-test-slave-69rg krb5kdc[17358](info): commencing operation
krb5kdc: starting...
W20260806 17:02:53.201733 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:02:53 dist-test-slave-69rg krb5kdc[17358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:02:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:02:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.126.keytab.
Entry for principal oryx/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.126.keytab.
I20260806 17:02:53.342150 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:40637
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.126.keytab
--principal=oryx/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:40637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:53.456910 17371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:53.457247 17371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:53.457337 17371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:53.461402 17371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:53.461503 17371 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 17:02:53.461539 17371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:53.461583 17371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:53.461620 17371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:53.461652 17371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:02:53.467070 17371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:40637
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.126.keytab
--principal=oryx/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:40637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:53.468181 17371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:53.469244 17371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:53.477767 17380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:53.477726 17378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:53.477783 17377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:53.478173 17371 server_base.cc:1034] running on GCE node
I20260806 17:02:53.478829 17371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:53.479918 17371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:53.481083 17371 hybrid_clock.cc:648] HybridClock initialized: now 1786035773481063 us; error 35 us; skew 500 ppm
Aug 06 17:02:53 dist-test-slave-69rg krb5kdc[17358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035773, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:53.483996 17371 init.cc:377] Logged in from keytab as oryx/127.9.111.126@KRBTEST.COM (short username oryx)
I20260806 17:02:53.484874 17371 webserver.cc:460] Webserver started at http://127.9.111.126:44313/ using document root <none> and password file <none>
I20260806 17:02:53.485428 17371 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:53.485519 17371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:53.485746 17371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:53.487785 17371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "6cca8e0737954706b30ebace8f6c5c02"
format_stamp: "Formatted at 2026-08-06 17:02:53 on dist-test-slave-69rg"
server_key: "2bc7bfa7d04a7dc44b7d599afc125f15"
server_key_iv: "8213aa39d6f1b4938f19295cfa0bcbd6"
server_key_version: "encryptionkey@0"
I20260806 17:02:53.488373 17371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "6cca8e0737954706b30ebace8f6c5c02"
format_stamp: "Formatted at 2026-08-06 17:02:53 on dist-test-slave-69rg"
server_key: "2bc7bfa7d04a7dc44b7d599afc125f15"
server_key_iv: "8213aa39d6f1b4938f19295cfa0bcbd6"
server_key_version: "encryptionkey@0"
I20260806 17:02:53.492122 17371 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 17:02:53.494463 17387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:53.495673 17371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:02:53.495827 17371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "6cca8e0737954706b30ebace8f6c5c02"
format_stamp: "Formatted at 2026-08-06 17:02:53 on dist-test-slave-69rg"
server_key: "2bc7bfa7d04a7dc44b7d599afc125f15"
server_key_iv: "8213aa39d6f1b4938f19295cfa0bcbd6"
server_key_version: "encryptionkey@0"
I20260806 17:02:53.495942 17371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:53.506795 17371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:53.510162 17371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:53.510370 17371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:53.518260 17371 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:40637
I20260806 17:02:53.518286 17451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:40637 every 8 connection(s)
I20260806 17:02:53.519194 17371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:02:53.519569 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 17371
I20260806 17:02:53.519706 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:02:53.520006 9661 external_mini_cluster.cc:1428] Setting key 01ed958dfa6057ee615773b0d638753f
I20260806 17:02:53.522547 17452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:53.529034 17452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02: Bootstrap starting.
Aug 06 17:02:53 dist-test-slave-69rg krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM
I20260806 17:02:53.531965 17452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:53.533354 17452 log.cc:824] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:53.535501 17452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02: No bootstrap required, opened a new log
I20260806 17:02:53.535904 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:53.521447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33850 (local address 127.9.111.126:40637)
0806 17:02:53.521900 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:53.521919 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:53.521962 (+ 43us) server_negotiation.cc:407] Connection header received
0806 17:02:53.522817 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:53.522839 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:53.523240 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:53.523592 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:53.524274 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:53.525735 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:53.526620 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:53.527303 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:53.527467 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:53.530607 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:53.530616 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:53.530632 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:53.530664 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:53.532932 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:53.533396 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:53.533402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:53.533408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:53.533550 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:53.533866 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:53.533870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:53.533872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:53.534285 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:53.534510 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:53.534805 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 17:02:53.535122 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":115,"threads_started":1}
I20260806 17:02:53.539204 17452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca8e0737954706b30ebace8f6c5c02" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 40637 } }
I20260806 17:02:53.539445 17452 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:53.539521 17452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cca8e0737954706b30ebace8f6c5c02, State: Initialized, Role: FOLLOWER
I20260806 17:02:53.539975 17452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca8e0737954706b30ebace8f6c5c02" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 40637 } }
I20260806 17:02:53.540118 17452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:02:53.540185 17452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:02:53.540292 17452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:53.541312 17452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca8e0737954706b30ebace8f6c5c02" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 40637 } }
I20260806 17:02:53.541673 17452 leader_election.cc:302] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cca8e0737954706b30ebace8f6c5c02; no voters:
I20260806 17:02:53.542078 17452 leader_election.cc:288] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:02:53.542243 17457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:53.542498 17457 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02 [term 1 LEADER]: Becoming Leader. State: Replica: 6cca8e0737954706b30ebace8f6c5c02, State: Running, Role: LEADER
I20260806 17:02:53.542830 17457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca8e0737954706b30ebace8f6c5c02" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 40637 } }
I20260806 17:02:53.543426 17452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:53.544711 17459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cca8e0737954706b30ebace8f6c5c02. Latest consensus state: current_term: 1 leader_uuid: "6cca8e0737954706b30ebace8f6c5c02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca8e0737954706b30ebace8f6c5c02" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 40637 } } }
I20260806 17:02:53.544716 17458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cca8e0737954706b30ebace8f6c5c02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca8e0737954706b30ebace8f6c5c02" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 40637 } } }
I20260806 17:02:53.544867 17458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:53.544866 17459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02 [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:53.545333 17466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:53.548729 17466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:02:53.555068 17466 catalog_manager.cc:1347] Generated new cluster ID: 6a397eaf333747d4b115545463a3ac8e
I20260806 17:02:53.555198 17466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:53.570623 17466 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:02:53.571651 17466 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:53.576802 17466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02: Generated new TSK 0
I20260806 17:02:53.577664 17466 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.65.keytab.
Entry for principal oryx/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.65.keytab.
I20260806 17:02:53.625380 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:40637
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.65.keytab
--principal=oryx/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:53.743124 17478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:53.743577 17478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:53.743669 17478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:53.748479 17478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:53.748591 17478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:53.748665 17478 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 17:02:53.748732 17478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:02:53.754355 17478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.65.keytab
--principal=oryx/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.111.126:40637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:53.755573 17478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:53.756655 17478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:53.766391 17484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:53.766396 17483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:53.766436 17486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:53.766676 17478 server_base.cc:1034] running on GCE node
I20260806 17:02:53.767258 17478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:53.767948 17478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:53.769122 17478 hybrid_clock.cc:648] HybridClock initialized: now 1786035773769085 us; error 47 us; skew 500 ppm
Aug 06 17:02:53 dist-test-slave-69rg krb5kdc[17358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035773, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:53.772054 17478 init.cc:377] Logged in from keytab as oryx/127.9.111.65@KRBTEST.COM (short username oryx)
I20260806 17:02:53.772967 17478 webserver.cc:460] Webserver started at http://127.9.111.65:39207/ using document root <none> and password file <none>
I20260806 17:02:53.773576 17478 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:53.773639 17478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:53.773934 17478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:53.776060 17478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "ab8197e88b3a413ab3841456ccd766b3"
format_stamp: "Formatted at 2026-08-06 17:02:53 on dist-test-slave-69rg"
server_key: "06aaf7e5ed720b2970955fff8d14cbfe"
server_key_iv: "43747709d22045d7cecdd91c503e5272"
server_key_version: "encryptionkey@0"
I20260806 17:02:53.776700 17478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "ab8197e88b3a413ab3841456ccd766b3"
format_stamp: "Formatted at 2026-08-06 17:02:53 on dist-test-slave-69rg"
server_key: "06aaf7e5ed720b2970955fff8d14cbfe"
server_key_iv: "43747709d22045d7cecdd91c503e5272"
server_key_version: "encryptionkey@0"
I20260806 17:02:53.780670 17478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 17:02:53.783217 17493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:53.784497 17478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 17:02:53.784649 17478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "ab8197e88b3a413ab3841456ccd766b3"
format_stamp: "Formatted at 2026-08-06 17:02:53 on dist-test-slave-69rg"
server_key: "06aaf7e5ed720b2970955fff8d14cbfe"
server_key_iv: "43747709d22045d7cecdd91c503e5272"
server_key_version: "encryptionkey@0"
I20260806 17:02:53.784776 17478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:53.827936 17478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:53.831108 17478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:53.831337 17478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:53.832806 17478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:53.832894 17478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:53.832986 17478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:53.833045 17478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 17:02:53 dist-test-slave-69rg krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035773, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM
I20260806 17:02:53.844880 17478 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:39521
I20260806 17:02:53.844959 17611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:39521 every 8 connection(s)
I20260806 17:02:53.846098 17478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:02:53.848459 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:53.833758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:49103 (local address 127.9.111.126:40637)
0806 17:02:53.833927 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:53.833931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:53.835015 (+ 1084us) server_negotiation.cc:407] Connection header received
0806 17:02:53.835987 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:53.835994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:53.836066 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:53.836170 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:53.837770 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:53.838389 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:53.839572 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:53.839824 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:53.839946 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:53.844088 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:53.844095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:53.844101 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:53.844144 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:53.846022 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:53.846787 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:53.846791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:53.846794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:53.846898 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:53.847317 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:53.847320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:53.847321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:53.847530 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:53.847621 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:53.848201 (+ 580us) server_negotiation.cc:299] Negotiation successful
0806 17:02:53.848331 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 17:02:53.849362 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:53.834377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:40637 (local address 127.9.111.65:49103)
0806 17:02:53.834866 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:53.834911 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:53.835714 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:53.836434 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:53.836445 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:53.836947 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:53.837611 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:53.837634 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:53.838535 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:53.838538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:53.839432 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:53.839442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:53.840017 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:53.840022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:53.840230 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:53.840930 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:53.840954 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:53.843870 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:53.846215 (+ 2345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:53.846221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:53.846236 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:53.846626 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:53.847013 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:53.847016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:53.847018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:53.847211 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:53.847693 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:53.847698 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:53.847965 (+ 267us) client_negotiation.cc:769] Sending connection context
0806 17:02:53.848139 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 17:02:53.848441 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":135,"threads_started":1}
I20260806 17:02:53.850945 17612 heartbeater.cc:347] Connected to a master server at 127.9.111.126:40637
I20260806 17:02:53.851140 17612 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:53.851745 17612 heartbeater.cc:511] Master 127.9.111.126:40637 requested a full tablet report, sending...
I20260806 17:02:53.853560 17403 ts_manager.cc:194] Registered new tserver with Master: ab8197e88b3a413ab3841456ccd766b3 (127.9.111.65:39521)
I20260806 17:02:53.854708 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 17478
I20260806 17:02:53.854832 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:02:53.855064 17403 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:49103
I20260806 17:02:53.855185 9661 external_mini_cluster.cc:1428] Setting key 2c80ddcfc75821035abf75d5a73ee1d4
I20260806 17:02:53.861485 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:53.853229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:52871 (local address 127.9.111.126:40637)
0806 17:02:53.853386 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:53.853394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:53.853612 (+ 218us) server_negotiation.cc:407] Connection header received
0806 17:02:53.853807 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:53.853810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:53.853889 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:53.853981 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:53.854463 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:53.855086 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:53.856207 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:53.856531 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:53.856674 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:53.858160 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:53.858163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:53.858165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:53.858188 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:53.859688 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:53.860240 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:53.860243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:53.860245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:53.860322 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:53.860599 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:53.860601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:53.860602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:53.860798 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:53.860916 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:53.861184 (+ 268us) server_negotiation.cc:299] Negotiation successful
0806 17:02:53.861300 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 17:02:53.861497 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:53.853209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:40637 (local address 127.9.111.65:52871)
0806 17:02:53.853524 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:53.853537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:53.853701 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:53.853998 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:53.854001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:53.854225 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:53.854327 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:53.854333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:53.855291 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:53.855295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:53.856085 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:53.856095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:53.856671 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:53.856674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:53.856774 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:53.857392 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:53.857413 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:53.857985 (+ 572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:53.859807 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:53.859814 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:53.859816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:53.860135 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:53.860415 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:53.860418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:53.860420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:53.860510 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:53.860905 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:53.860908 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:53.861014 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 17:02:53.861127 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 17:02:53.861302 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":134,"threads_started":1}
WARNING: no policy specified for oryx/127.9.111.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.111.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.66.keytab.
Entry for principal oryx/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.66.keytab.
I20260806 17:02:53.888513 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:40637
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.66.keytab
--principal=oryx/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:54.001689 17621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:54.002071 17621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:54.002161 17621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:54.006381 17621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:54.006476 17621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:54.006548 17621 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 17:02:54.006632 17621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:02:54.012365 17621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.66.keytab
--principal=oryx/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.111.126:40637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:54.013622 17621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:54.014736 17621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:54.024500 17627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:54.024471 17629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:54.024994 17621 server_base.cc:1034] running on GCE node
W20260806 17:02:54.025228 17626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:54.026276 17621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:54.026983 17621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:54.028180 17621 hybrid_clock.cc:648] HybridClock initialized: now 1786035774028164 us; error 30 us; skew 500 ppm
Aug 06 17:02:54 dist-test-slave-69rg krb5kdc[17358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035774, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:54.031154 17621 init.cc:377] Logged in from keytab as oryx/127.9.111.66@KRBTEST.COM (short username oryx)
I20260806 17:02:54.032151 17621 webserver.cc:460] Webserver started at http://127.9.111.66:40281/ using document root <none> and password file <none>
I20260806 17:02:54.032734 17621 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:54.032829 17621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:54.033063 17621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:54.035215 17621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "b15c4a303d934243b47be0a30e0cd2d5"
format_stamp: "Formatted at 2026-08-06 17:02:54 on dist-test-slave-69rg"
server_key: "a2a1d909a3cd23e66b4e6751433c4c29"
server_key_iv: "7c7bc826077adb346c14fcdb0344b18d"
server_key_version: "encryptionkey@0"
I20260806 17:02:54.035826 17621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "b15c4a303d934243b47be0a30e0cd2d5"
format_stamp: "Formatted at 2026-08-06 17:02:54 on dist-test-slave-69rg"
server_key: "a2a1d909a3cd23e66b4e6751433c4c29"
server_key_iv: "7c7bc826077adb346c14fcdb0344b18d"
server_key_version: "encryptionkey@0"
I20260806 17:02:54.039599 17621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 17:02:54.042088 17636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:54.043431 17621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:02:54.043599 17621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "b15c4a303d934243b47be0a30e0cd2d5"
format_stamp: "Formatted at 2026-08-06 17:02:54 on dist-test-slave-69rg"
server_key: "a2a1d909a3cd23e66b4e6751433c4c29"
server_key_iv: "7c7bc826077adb346c14fcdb0344b18d"
server_key_version: "encryptionkey@0"
I20260806 17:02:54.043759 17621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:54.062879 17621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:54.067427 17621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:54.067664 17621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:54.069566 17621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:54.069648 17621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:54.069720 17621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:54.069751 17621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 17:02:54 dist-test-slave-69rg krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035774, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM
I20260806 17:02:54.081418 17621 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:43187
I20260806 17:02:54.081535 17753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:43187 every 8 connection(s)
I20260806 17:02:54.082448 17621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:02:54.085525 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.070473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:36841 (local address 127.9.111.126:40637)
0806 17:02:54.070603 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:54.070615 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:54.071688 (+ 1073us) server_negotiation.cc:407] Connection header received
0806 17:02:54.072901 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:54.072904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:54.072966 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:54.073065 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:54.074873 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.075725 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.077888 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.078226 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.078319 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.081754 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:54.081758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:54.081760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:54.081787 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:54.083597 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:54.084098 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:54.084101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:54.084102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:54.084196 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:54.084532 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:54.084535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:54.084536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:54.084731 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:54.084810 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:54.085292 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 17:02:54.085405 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 17:02:54.086061 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 17621
I20260806 17:02:54.086187 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:02:54.086460 17646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.071016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:40637 (local address 127.9.111.66:36841)
0806 17:02:54.071538 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:54.071581 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:54.072620 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:54.073301 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:54.073314 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:54.073857 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:54.074631 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.074656 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.075878 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.075884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.077725 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.077738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.078397 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.078403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.078632 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.079363 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:54.079387 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:54.081528 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:54.083708 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:54.083712 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:54.083724 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:54.083983 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:54.084314 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:54.084317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:54.084319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:54.084450 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:54.084832 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:54.084837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:54.085059 (+ 222us) client_negotiation.cc:769] Sending connection context
0806 17:02:54.085212 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 17:02:54.085457 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":145,"threads_started":1}
I20260806 17:02:54.086489 9661 external_mini_cluster.cc:1428] Setting key 888bf32389e709cc41644d7b69166603
I20260806 17:02:54.087718 17754 heartbeater.cc:347] Connected to a master server at 127.9.111.126:40637
I20260806 17:02:54.087852 17754 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:54.088407 17754 heartbeater.cc:511] Master 127.9.111.126:40637 requested a full tablet report, sending...
I20260806 17:02:54.089578 17404 ts_manager.cc:194] Registered new tserver with Master: b15c4a303d934243b47be0a30e0cd2d5 (127.9.111.66:43187)
I20260806 17:02:54.090441 17404 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:36841
I20260806 17:02:54.098431 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.090103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:56803 (local address 127.9.111.126:40637)
0806 17:02:54.090252 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:54.090257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:54.090966 (+ 709us) server_negotiation.cc:407] Connection header received
0806 17:02:54.091073 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:54.091078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:54.091139 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:54.091236 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:54.091755 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.092536 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.093358 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.093582 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.093692 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.095036 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:54.095039 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:54.095042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:54.095065 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:54.096530 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:54.097104 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:54.097107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:54.097109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:54.097201 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:54.097530 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:54.097532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:54.097533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:54.097752 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:54.097886 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:54.098186 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 17:02:54.098287 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 17:02:54.098562 17760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.090220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:40637 (local address 127.9.111.66:56803)
0806 17:02:54.090864 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:54.090880 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:54.090979 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:54.091266 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:54.091269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:54.091512 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:54.091626 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.091634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.092662 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.092665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.093232 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.093241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.093697 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.093708 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.093847 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.094359 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:54.094379 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:54.094885 (+ 506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:54.096650 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:54.096655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:54.096660 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:54.096963 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:54.097308 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:54.097314 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:54.097318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:54.097427 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:54.097876 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:54.097882 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:54.098001 (+ 119us) client_negotiation.cc:769] Sending connection context
0806 17:02:54.098139 (+ 138us) client_negotiation.cc:240] Negotiation successful
0806 17:02:54.098314 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for oryx/127.9.111.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.111.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.67.keytab.
Entry for principal oryx/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.67.keytab.
I20260806 17:02:54.121555 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:40637
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.67.keytab
--principal=oryx/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:54.236591 17763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:54.236879 17763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:54.236928 17763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:54.241053 17763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:54.241137 17763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:54.241178 17763 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 17:02:54.241252 17763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:02:54.246981 17763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.67.keytab
--principal=oryx/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.111.126:40637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:54.248319 17763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:54.249439 17763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:54.259227 17769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:54.259250 17768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:54.259217 17771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:54.259728 17763 server_base.cc:1034] running on GCE node
I20260806 17:02:54.260078 17763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:54.260664 17763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:54.261881 17763 hybrid_clock.cc:648] HybridClock initialized: now 1786035774261862 us; error 27 us; skew 500 ppm
Aug 06 17:02:54 dist-test-slave-69rg krb5kdc[17358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035774, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:54.264750 17763 init.cc:377] Logged in from keytab as oryx/127.9.111.67@KRBTEST.COM (short username oryx)
I20260806 17:02:54.265578 17763 webserver.cc:460] Webserver started at http://127.9.111.67:37975/ using document root <none> and password file <none>
I20260806 17:02:54.266217 17763 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:54.266283 17763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:54.266454 17763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:54.268558 17763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "45e9aff505ca4498a8128f450b898170"
format_stamp: "Formatted at 2026-08-06 17:02:54 on dist-test-slave-69rg"
server_key: "8a1fc78ade0c1f0d8be28fd505726656"
server_key_iv: "3cc7654b4669cc7896d67b32eef2bde3"
server_key_version: "encryptionkey@0"
I20260806 17:02:54.269080 17763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "45e9aff505ca4498a8128f450b898170"
format_stamp: "Formatted at 2026-08-06 17:02:54 on dist-test-slave-69rg"
server_key: "8a1fc78ade0c1f0d8be28fd505726656"
server_key_iv: "3cc7654b4669cc7896d67b32eef2bde3"
server_key_version: "encryptionkey@0"
I20260806 17:02:54.272862 17763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260806 17:02:54.275530 17778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:54.276886 17763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 17:02:54.277038 17763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "45e9aff505ca4498a8128f450b898170"
format_stamp: "Formatted at 2026-08-06 17:02:54 on dist-test-slave-69rg"
server_key: "8a1fc78ade0c1f0d8be28fd505726656"
server_key_iv: "3cc7654b4669cc7896d67b32eef2bde3"
server_key_version: "encryptionkey@0"
I20260806 17:02:54.277166 17763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:54.298161 17763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:54.301977 17763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:54.302219 17763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:54.303722 17763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:54.303803 17763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:54.303877 17763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:54.303920 17763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 17:02:54 dist-test-slave-69rg krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035774, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM
I20260806 17:02:54.315728 17763 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:42695
I20260806 17:02:54.315824 17893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:42695 every 8 connection(s)
I20260806 17:02:54.316649 17763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:02:54.319198 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 17763
I20260806 17:02:54.319352 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:02:54.319747 9661 external_mini_cluster.cc:1428] Setting key a035eda0f4263527a1c8a5ff2f584c7c
I20260806 17:02:54.320009 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.304973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:46975 (local address 127.9.111.126:40637)
0806 17:02:54.305109 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:54.305114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:54.306277 (+ 1163us) server_negotiation.cc:407] Connection header received
0806 17:02:54.307486 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:54.307490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:54.307552 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:54.307662 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:54.309207 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.310072 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.311389 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.311763 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.311871 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.315739 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:54.315743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:54.315746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:54.315783 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:54.317656 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:54.318241 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:54.318244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:54.318245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:54.318334 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:54.318671 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:54.318674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:54.318675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:54.318870 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:54.318946 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:54.319704 (+ 758us) server_negotiation.cc:299] Negotiation successful
0806 17:02:54.319853 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 17:02:54.320988 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.305502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:40637 (local address 127.9.111.67:46975)
0806 17:02:54.306062 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:54.306163 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:54.307238 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:54.307833 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:54.307841 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:54.308290 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:54.309017 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.309037 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.310217 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.310220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.311252 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.311262 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.311918 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.311923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.312087 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.313471 (+ 1384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:54.313501 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:54.315531 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:54.317769 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:54.317773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:54.317784 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:54.318117 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:54.318420 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:54.318422 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:54.318423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:54.318548 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:54.318983 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:54.318991 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:54.319389 (+ 398us) client_negotiation.cc:769] Sending connection context
0806 17:02:54.319610 (+ 221us) client_negotiation.cc:240] Negotiation successful
0806 17:02:54.319955 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":127,"threads_started":1}
I20260806 17:02:54.322283 17894 heartbeater.cc:347] Connected to a master server at 127.9.111.126:40637
I20260806 17:02:54.322441 17894 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:54.322947 17894 heartbeater.cc:511] Master 127.9.111.126:40637 requested a full tablet report, sending...
I20260806 17:02:54.323959 17403 ts_manager.cc:194] Registered new tserver with Master: 45e9aff505ca4498a8128f450b898170 (127.9.111.67:42695)
I20260806 17:02:54.324708 17403 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:46975
I20260806 17:02:54.331521 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.324506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:40637 (local address 127.9.111.67:56841)
0806 17:02:54.324810 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:54.324826 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:54.324927 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:54.325168 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:54.325171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:54.325398 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:54.325497 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.325503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.326330 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.326333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.326888 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.326896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.327355 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.327359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.327456 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.327955 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:54.327966 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:54.328288 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:54.329981 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:54.329986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:54.329988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:54.330190 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:54.330483 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:54.330487 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:54.330489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:54.330574 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:54.330954 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:54.330957 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:54.331058 (+ 101us) client_negotiation.cc:769] Sending connection context
0806 17:02:54.331165 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 17:02:54.331316 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":112,"threads_started":1}
I20260806 17:02:54.331533 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.324575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:56841 (local address 127.9.111.126:40637)
0806 17:02:54.324744 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:54.324749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:54.324883 (+ 134us) server_negotiation.cc:407] Connection header received
0806 17:02:54.325014 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:54.325017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:54.325064 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:54.325152 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:54.325616 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.326202 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.327014 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.327237 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.327342 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.328422 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:54.328426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:54.328428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:54.328459 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:54.329864 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:54.330301 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:54.330304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:54.330306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:54.330388 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:54.330666 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:54.330670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:54.330672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:54.330857 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:54.330966 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:54.331279 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 17:02:54.331399 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 17:02:54.334174 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 17:02:54 dist-test-slave-69rg krb5kdc[17358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 17:02:54 dist-test-slave-69rg krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:02:54 dist-test-slave-69rg krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
W20260806 17:02:54.360365 17455 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:02:54.353884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.9.111.126:40637)
0806 17:02:54.354016 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:54.354019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:54.354032 (+ 13us) server_negotiation.cc:407] Connection header received
0806 17:02:54.354075 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:54.354078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:54.354125 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:54.354224 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:54.355009 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.355585 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.356496 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.356722 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.356835 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.360202 (+ 3367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33852: BlockingRecv error: recv got EOF from 127.0.0.1:33852 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 06 17:02:54 dist-test-slave-69rg krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM
I20260806 17:02:54.370112 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.361989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33868 (local address 127.9.111.126:40637)
0806 17:02:54.362141 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:54.362144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:54.362222 (+ 78us) server_negotiation.cc:407] Connection header received
0806 17:02:54.362311 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:54.362313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:54.362353 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:54.362421 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:54.363003 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.363554 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.364318 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.364523 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.364605 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.367216 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:54.367219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:54.367221 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:54.367247 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:54.368558 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:54.368944 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:54.368946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:54.368947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:54.369024 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:54.369251 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:54.369254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:54.369255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:54.369447 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:54.369551 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:54.369798 (+ 247us) server_negotiation.cc:299] Negotiation successful
0806 17:02:54.369987 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 17:02:54.373284 17403 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:02:54.376384 17403 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:02:54.391592 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.385477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:42695 (local address 127.0.0.1:40972)
0806 17:02:54.386289 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:54.386302 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:54.386378 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:54.387001 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:54.387004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:54.387023 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:54.387126 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.387134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.388381 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.388386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.390179 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.390187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.391008 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.391012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.391120 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.391160 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 17:02:54.391279 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 17:02:54.391413 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":107,"thread_start_us":83,"threads_started":1}
I20260806 17:02:54.391587 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.385864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40972 (local address 127.9.111.67:42695)
0806 17:02:54.386357 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:54.386369 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:54.386395 (+ 26us) server_negotiation.cc:407] Connection header received
0806 17:02:54.386478 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:54.386483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:54.386695 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:54.386846 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:54.387292 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.388263 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.390316 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.390878 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.391008 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.391189 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:54.391310 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 17:02:54.391410 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":111,"threads_started":1}
I20260806 17:02:54.391647 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.385237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:39521 (local address 127.0.0.1:56312)
0806 17:02:54.385871 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:54.385924 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:54.386068 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:54.386872 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:54.386879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:54.386901 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:54.387064 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.387072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.388569 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.388573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.390152 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.390163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.391173 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.391177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.391278 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.391309 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 17:02:54.391366 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 17:02:54.391451 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":114,"spinlock_wait_cycles":4480,"thread_start_us":105,"threads_started":1}
I20260806 17:02:54.391985 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.385225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42768 (local address 127.9.111.66:43187)
0806 17:02:54.386633 (+ 1408us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:54.386644 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:54.386664 (+ 20us) server_negotiation.cc:407] Connection header received
0806 17:02:54.386736 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:54.386745 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:54.386919 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:54.387057 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:54.387370 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.388561 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.390164 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.391026 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.391149 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.391329 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:54.391718 (+ 389us) server_negotiation.cc:299] Negotiation successful
0806 17:02:54.391825 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1300,"thread_start_us":134,"threads_started":1}
I20260806 17:02:54.392442 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.385861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56312 (local address 127.9.111.65:39521)
0806 17:02:54.386385 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:54.386392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:54.386415 (+ 23us) server_negotiation.cc:407] Connection header received
0806 17:02:54.386489 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:54.386497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:54.386698 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:54.386849 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:54.387207 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.388474 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.390288 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.391031 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.391991 (+ 960us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.392153 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:54.392204 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 17:02:54.392309 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":97,"threads_started":1}
I20260806 17:02:54.392709 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.385058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:43187 (local address 127.0.0.1:42768)
0806 17:02:54.385871 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:54.385924 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:54.386068 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:54.387040 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:54.387043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:54.387061 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:54.387215 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.387225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.388676 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.388683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.390036 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.390046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.391389 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.391393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.391484 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.391509 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 17:02:54.391574 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 17:02:54.391634 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":45,"spinlock_wait_cycles":28160,"thread_start_us":96,"threads_started":1}
I20260806 17:02:54.395185 17546 tablet_service.cc:1467] Processing CreateTablet for tablet ca2809173d464eee8f1d5f84f723f8fd (DEFAULT_TABLE table=test-table [id=1ea08de7420142f39390c5b89cc07a32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:54.395337 17828 tablet_service.cc:1467] Processing CreateTablet for tablet ca2809173d464eee8f1d5f84f723f8fd (DEFAULT_TABLE table=test-table [id=1ea08de7420142f39390c5b89cc07a32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:54.395337 17687 tablet_service.cc:1467] Processing CreateTablet for tablet ca2809173d464eee8f1d5f84f723f8fd (DEFAULT_TABLE table=test-table [id=1ea08de7420142f39390c5b89cc07a32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:54.396260 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca2809173d464eee8f1d5f84f723f8fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:54.396345 17687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca2809173d464eee8f1d5f84f723f8fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:54.396345 17828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca2809173d464eee8f1d5f84f723f8fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:54.403373 17923 tablet_bootstrap.cc:492] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3: Bootstrap starting.
I20260806 17:02:54.404184 17925 tablet_bootstrap.cc:492] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170: Bootstrap starting.
I20260806 17:02:54.405930 17923 tablet_bootstrap.cc:654] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:54.406169 17925 tablet_bootstrap.cc:654] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:54.406188 17924 tablet_bootstrap.cc:492] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5: Bootstrap starting.
I20260806 17:02:54.406884 17923 log.cc:824] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:54.406884 17925 log.cc:824] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:54.408099 17924 tablet_bootstrap.cc:654] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:54.408790 17924 log.cc:824] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:54.409101 17925 tablet_bootstrap.cc:492] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170: No bootstrap required, opened a new log
I20260806 17:02:54.409102 17923 tablet_bootstrap.cc:492] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3: No bootstrap required, opened a new log
I20260806 17:02:54.409299 17925 ts_tablet_manager.cc:1397] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 17:02:54.409667 17923 ts_tablet_manager.cc:1397] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 17:02:54.413594 17925 raft_consensus.cc:348] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:54.413949 17925 raft_consensus.cc:374] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:54.414013 17925 raft_consensus.cc:729] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e9aff505ca4498a8128f450b898170, State: Initialized, Role: FOLLOWER
I20260806 17:02:54.414263 17923 raft_consensus.cc:348] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:54.414531 17923 raft_consensus.cc:374] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:54.414587 17923 raft_consensus.cc:729] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab8197e88b3a413ab3841456ccd766b3, State: Initialized, Role: FOLLOWER
I20260806 17:02:54.414505 17925 consensus_queue.cc:260] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:54.415010 17923 consensus_queue.cc:260] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:54.415678 17612 heartbeater.cc:503] Master 127.9.111.126:40637 was elected leader, sending a full tablet report...
I20260806 17:02:54.415896 17923 ts_tablet_manager.cc:1428] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260806 17:02:54.415903 17924 tablet_bootstrap.cc:492] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5: No bootstrap required, opened a new log
I20260806 17:02:54.416074 17925 ts_tablet_manager.cc:1428] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 17:02:54.416109 17924 ts_tablet_manager.cc:1397] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260806 17:02:54.416381 17894 heartbeater.cc:503] Master 127.9.111.126:40637 was elected leader, sending a full tablet report...
I20260806 17:02:54.419972 17924 raft_consensus.cc:348] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:54.420253 17924 raft_consensus.cc:374] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:54.420333 17924 raft_consensus.cc:729] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15c4a303d934243b47be0a30e0cd2d5, State: Initialized, Role: FOLLOWER
I20260806 17:02:54.420800 17924 consensus_queue.cc:260] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:54.421505 17754 heartbeater.cc:503] Master 127.9.111.126:40637 was elected leader, sending a full tablet report...
I20260806 17:02:54.421850 17924 ts_tablet_manager.cc:1428] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260806 17:02:54.531122 17929 raft_consensus.cc:482] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:54.531339 17929 raft_consensus.cc:504] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:54.532892 17929 leader_election.cc:288] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b15c4a303d934243b47be0a30e0cd2d5 (127.9.111.66:43187), ab8197e88b3a413ab3841456ccd766b3 (127.9.111.65:39521)
I20260806 17:02:54.537562 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.533623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:53677 (local address 127.9.111.65:39521)
0806 17:02:54.533854 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:54.533860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:54.533877 (+ 17us) server_negotiation.cc:407] Connection header received
0806 17:02:54.533936 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:54.533940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:54.533989 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:54.534077 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:54.534395 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.535191 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.536281 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.536715 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.536858 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.536924 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:54.537371 (+ 447us) server_negotiation.cc:299] Negotiation successful
0806 17:02:54.537447 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 17:02:54.537586 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.533258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:39521 (local address 127.9.111.67:53677)
0806 17:02:54.533609 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:54.533621 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:54.533708 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:54.534106 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:54.534109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:54.534144 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:54.534274 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.534280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.535314 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.535317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.536142 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.536151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.536851 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.536855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.537226 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.537246 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 17:02:54.537344 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 17:02:54.537420 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":1,"thread_start_us":134,"threads_started":1}
I20260806 17:02:54.537859 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.534601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:37077 (local address 127.9.111.66:43187)
0806 17:02:54.534741 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:54.534754 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:54.534770 (+ 16us) server_negotiation.cc:407] Connection header received
0806 17:02:54.534823 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:54.534826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:54.534879 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:54.534953 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:54.535313 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.535812 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.536904 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.537298 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.537424 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.537510 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:54.537651 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 17:02:54.537721 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 17:02:54.537874 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.533129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:43187 (local address 127.9.111.67:37077)
0806 17:02:54.533251 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:54.533267 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:54.533368 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:54.535071 (+ 1703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:54.535074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:54.535091 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:54.535188 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.535192 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.535931 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.535936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.536745 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.536778 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.537417 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.537421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.537523 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.537551 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 17:02:54.537629 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 17:02:54.537705 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":53}
I20260806 17:02:54.538414 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca2809173d464eee8f1d5f84f723f8fd" candidate_uuid: "45e9aff505ca4498a8128f450b898170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab8197e88b3a413ab3841456ccd766b3" is_pre_election: true
I20260806 17:02:54.538498 17707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca2809173d464eee8f1d5f84f723f8fd" candidate_uuid: "45e9aff505ca4498a8128f450b898170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15c4a303d934243b47be0a30e0cd2d5" is_pre_election: true
I20260806 17:02:54.538772 17566 raft_consensus.cc:2454] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45e9aff505ca4498a8128f450b898170 in term 0.
I20260806 17:02:54.538791 17707 raft_consensus.cc:2454] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45e9aff505ca4498a8128f450b898170 in term 0.
I20260806 17:02:54.539296 17781 leader_election.cc:302] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 45e9aff505ca4498a8128f450b898170, ab8197e88b3a413ab3841456ccd766b3, b15c4a303d934243b47be0a30e0cd2d5; no voters:
I20260806 17:02:54.539583 17929 raft_consensus.cc:2781] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:02:54.539659 17929 raft_consensus.cc:482] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:02:54.539690 17929 raft_consensus.cc:3037] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:54.540786 17929 raft_consensus.cc:504] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:54.541162 17929 leader_election.cc:288] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [CANDIDATE]: Term 1 election: Requested vote from peers b15c4a303d934243b47be0a30e0cd2d5 (127.9.111.66:43187), ab8197e88b3a413ab3841456ccd766b3 (127.9.111.65:39521)
I20260806 17:02:54.541556 17707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca2809173d464eee8f1d5f84f723f8fd" candidate_uuid: "45e9aff505ca4498a8128f450b898170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15c4a303d934243b47be0a30e0cd2d5"
I20260806 17:02:54.541535 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca2809173d464eee8f1d5f84f723f8fd" candidate_uuid: "45e9aff505ca4498a8128f450b898170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab8197e88b3a413ab3841456ccd766b3"
I20260806 17:02:54.541692 17707 raft_consensus.cc:3037] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:54.541695 17566 raft_consensus.cc:3037] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:54.542928 17707 raft_consensus.cc:2454] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45e9aff505ca4498a8128f450b898170 in term 1.
I20260806 17:02:54.543223 17781 leader_election.cc:302] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45e9aff505ca4498a8128f450b898170, b15c4a303d934243b47be0a30e0cd2d5; no voters:
I20260806 17:02:54.543391 17929 raft_consensus.cc:2781] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:54.543423 17566 raft_consensus.cc:2454] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45e9aff505ca4498a8128f450b898170 in term 1.
I20260806 17:02:54.543690 17929 raft_consensus.cc:686] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [term 1 LEADER]: Becoming Leader. State: Replica: 45e9aff505ca4498a8128f450b898170, State: Running, Role: LEADER
I20260806 17:02:54.544078 17929 consensus_queue.cc:237] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:54.547557 17403 catalog_manager.cc:5495] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45e9aff505ca4498a8128f450b898170 (127.9.111.67). New cstate: current_term: 1 leader_uuid: "45e9aff505ca4498a8128f450b898170" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } health_report { overall_health: HEALTHY } } }
W20260806 17:02:54.567828 17895 tablet.cc:2369] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 17:02:54.583892 17755 tablet.cc:2369] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:02:54.588882 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.585005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40974 (local address 127.9.111.67:42695)
0806 17:02:54.585133 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:54.585137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:54.585151 (+ 14us) server_negotiation.cc:407] Connection header received
0806 17:02:54.585334 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:54.585336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:54.585395 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:54.585482 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:54.585703 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.586238 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.587102 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.587408 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.587516 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.587839 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:54.588429 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:54.588518 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:54.588645 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 17:02:54.588708 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 17:02:54.594883 17566 raft_consensus.cc:1261] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3 [term 1 FOLLOWER]: Refusing update from remote peer 45e9aff505ca4498a8128f450b898170: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:54.594882 17707 raft_consensus.cc:1261] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5 [term 1 FOLLOWER]: Refusing update from remote peer 45e9aff505ca4498a8128f450b898170: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:54.595595 17933 consensus_queue.cc:1035] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:54.595782 17929 consensus_queue.cc:1035] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 17:02:54.597648 17613 tablet.cc:2369] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:02:54.604322 17937 mvcc.cc:205] Tried to move back new op lower bound from 7315602532733075456 to 7315602532535390208. Current Snapshot: MvccSnapshot[applied={T|T < 7315602532733075456}]
I20260806 17:02:54.604911 17938 mvcc.cc:205] Tried to move back new op lower bound from 7315602532733075456 to 7315602532535390208. Current Snapshot: MvccSnapshot[applied={T|T < 7315602532733075456}]
I20260806 17:02:54.612295 17405 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:54.612479 17405 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:54.615375 17405 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02: Sending DeleteTablet for 3 replicas of tablet ca2809173d464eee8f1d5f84f723f8fd
I20260806 17:02:54.616247 17687 tablet_service.cc:1514] Processing DeleteTablet for tablet ca2809173d464eee8f1d5f84f723f8fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:54 UTC) from {username='oryx', principal='oryx/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:42768
I20260806 17:02:54.616441 17828 tablet_service.cc:1514] Processing DeleteTablet for tablet ca2809173d464eee8f1d5f84f723f8fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:54 UTC) from {username='oryx', principal='oryx/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:40972
I20260806 17:02:54.616465 17546 tablet_service.cc:1514] Processing DeleteTablet for tablet ca2809173d464eee8f1d5f84f723f8fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:54 UTC) from {username='oryx', principal='oryx/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:56312
I20260806 17:02:54.616918 17947 tablet_replica.cc:330] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170: stopping tablet replica
I20260806 17:02:54.617080 17948 tablet_replica.cc:330] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3: stopping tablet replica
I20260806 17:02:54.617394 17947 raft_consensus.cc:2229] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:02:54.617410 17948 raft_consensus.cc:2229] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:54.617739 17948 raft_consensus.cc:2258] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:54.618040 17947 raft_consensus.cc:2258] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:54.618121 17405 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:02:54.618637 17405 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:02:54.619418 17948 ts_tablet_manager.cc:1905] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:54.620034 17947 ts_tablet_manager.cc:1905] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:54.622401 17948 ts_tablet_manager.cc:1918] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:02:54.622514 17948 log.cc:1196] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/wal/wals/ca2809173d464eee8f1d5f84f723f8fd
I20260806 17:02:54.622915 17948 ts_tablet_manager.cc:1939] T ca2809173d464eee8f1d5f84f723f8fd P ab8197e88b3a413ab3841456ccd766b3: Deleting consensus metadata
I20260806 17:02:54.623636 17947 ts_tablet_manager.cc:1918] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:02:54.623828 17947 log.cc:1196] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/ca2809173d464eee8f1d5f84f723f8fd
I20260806 17:02:54.624257 17389 catalog_manager.cc:4841] TS ab8197e88b3a413ab3841456ccd766b3 (127.9.111.65:39521): tablet ca2809173d464eee8f1d5f84f723f8fd (table test-table [id=1ea08de7420142f39390c5b89cc07a32]) successfully deleted
I20260806 17:02:54.624552 17947 ts_tablet_manager.cc:1939] T ca2809173d464eee8f1d5f84f723f8fd P 45e9aff505ca4498a8128f450b898170: Deleting consensus metadata
I20260806 17:02:54.624897 17546 tablet_service.cc:1467] Processing CreateTablet for tablet b444dd52de5d48a19249796aab3e1347 (DEFAULT_TABLE table=test-table [id=b94d8aca8abb4a34b287b914113ab4dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:54.625155 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b444dd52de5d48a19249796aab3e1347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:54.625135 17828 tablet_service.cc:1467] Processing CreateTablet for tablet b444dd52de5d48a19249796aab3e1347 (DEFAULT_TABLE table=test-table [id=b94d8aca8abb4a34b287b914113ab4dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:54.625434 17828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b444dd52de5d48a19249796aab3e1347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:54.627395 17923 tablet_bootstrap.cc:492] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3: Bootstrap starting.
I20260806 17:02:54.627517 17925 tablet_bootstrap.cc:492] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170: Bootstrap starting.
I20260806 17:02:54.628525 17925 tablet_bootstrap.cc:654] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:54.628558 17923 tablet_bootstrap.cc:654] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:54.629349 17391 catalog_manager.cc:4841] TS 45e9aff505ca4498a8128f450b898170 (127.9.111.67:42695): tablet ca2809173d464eee8f1d5f84f723f8fd (table test-table [id=1ea08de7420142f39390c5b89cc07a32]) successfully deleted
I20260806 17:02:54.630086 17925 tablet_bootstrap.cc:492] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170: No bootstrap required, opened a new log
I20260806 17:02:54.630110 17923 tablet_bootstrap.cc:492] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3: No bootstrap required, opened a new log
I20260806 17:02:54.630185 17923 ts_tablet_manager.cc:1397] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260806 17:02:54.630184 17925 ts_tablet_manager.cc:1397] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 17:02:54.630218 17687 tablet_service.cc:1467] Processing CreateTablet for tablet b444dd52de5d48a19249796aab3e1347 (DEFAULT_TABLE table=test-table [id=b94d8aca8abb4a34b287b914113ab4dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:54.630517 17687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b444dd52de5d48a19249796aab3e1347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:54.630738 17925 raft_consensus.cc:348] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:54.630860 17925 raft_consensus.cc:374] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:54.630903 17925 raft_consensus.cc:729] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e9aff505ca4498a8128f450b898170, State: Initialized, Role: FOLLOWER
I20260806 17:02:54.631000 17923 raft_consensus.cc:348] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:54.631106 17923 raft_consensus.cc:374] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:54.631134 17923 raft_consensus.cc:729] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab8197e88b3a413ab3841456ccd766b3, State: Initialized, Role: FOLLOWER
I20260806 17:02:54.631250 17923 consensus_queue.cc:260] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:54.631281 17925 consensus_queue.cc:260] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:54.631494 17923 ts_tablet_manager.cc:1428] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 17:02:54.631626 17946 tablet_replica.cc:330] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5: stopping tablet replica
I20260806 17:02:54.631839 17925 ts_tablet_manager.cc:1428] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 17:02:54.633101 17924 tablet_bootstrap.cc:492] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5: Bootstrap starting.
I20260806 17:02:54.634234 17924 tablet_bootstrap.cc:654] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:54.635468 17946 raft_consensus.cc:2229] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:54.635915 17946 raft_consensus.cc:2258] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:54.636677 17924 tablet_bootstrap.cc:492] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5: No bootstrap required, opened a new log
I20260806 17:02:54.636778 17924 ts_tablet_manager.cc:1397] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 17:02:54.637264 17924 raft_consensus.cc:348] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:54.637396 17924 raft_consensus.cc:374] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:54.637432 17924 raft_consensus.cc:729] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15c4a303d934243b47be0a30e0cd2d5, State: Initialized, Role: FOLLOWER
I20260806 17:02:54.637575 17946 ts_tablet_manager.cc:1905] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:54.637916 17924 consensus_queue.cc:260] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:54.638353 17924 ts_tablet_manager.cc:1428] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 17:02:54.640709 17946 ts_tablet_manager.cc:1918] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:02:54.640834 17946 log.cc:1196] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/ca2809173d464eee8f1d5f84f723f8fd
I20260806 17:02:54.641244 17946 ts_tablet_manager.cc:1939] T ca2809173d464eee8f1d5f84f723f8fd P b15c4a303d934243b47be0a30e0cd2d5: Deleting consensus metadata
I20260806 17:02:54.642575 17390 catalog_manager.cc:4841] TS b15c4a303d934243b47be0a30e0cd2d5 (127.9.111.66:43187): tablet ca2809173d464eee8f1d5f84f723f8fd (table test-table [id=1ea08de7420142f39390c5b89cc07a32]) successfully deleted
I20260806 17:02:54.915730 17930 raft_consensus.cc:482] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:54.915910 17930 raft_consensus.cc:504] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:54.917124 17930 leader_election.cc:288] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b15c4a303d934243b47be0a30e0cd2d5 (127.9.111.66:43187), 45e9aff505ca4498a8128f450b898170 (127.9.111.67:42695)
I20260806 17:02:54.921144 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.917487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:44871 (local address 127.9.111.66:43187)
0806 17:02:54.917647 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:54.917654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:54.917884 (+ 230us) server_negotiation.cc:407] Connection header received
0806 17:02:54.918059 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:54.918067 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:54.918123 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:54.918208 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:54.918456 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.918956 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.920092 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.920584 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.920711 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.920766 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:54.920951 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 17:02:54.921020 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 17:02:54.921160 17952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.917356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:43187 (local address 127.9.111.65:44871)
0806 17:02:54.917739 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:54.917762 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:54.917935 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:54.918219 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:54.918222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:54.918237 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:54.918345 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.918350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.919065 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.919071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.919966 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.919975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.920705 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.920709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.920815 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.920838 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 17:02:54.920934 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 17:02:54.921007 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":89,"threads_started":1}
I20260806 17:02:54.921438 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.917547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:36297 (local address 127.9.111.67:42695)
0806 17:02:54.917716 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:54.917721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:54.918011 (+ 290us) server_negotiation.cc:407] Connection header received
0806 17:02:54.918202 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:54.918206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:54.918259 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:54.918356 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:54.918659 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.919201 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.920322 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.920813 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.920953 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.921009 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:54.921222 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 17:02:54.921300 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 17:02:54.921455 17953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:54.917469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:42695 (local address 127.9.111.65:36297)
0806 17:02:54.917908 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:54.917922 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:54.918054 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:54.918369 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:54.918372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:54.918389 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:54.918525 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.918531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.919354 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.919358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.920179 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:54.920188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:54.920948 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:54.920952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:54.921072 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:54.921097 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 17:02:54.921202 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 17:02:54.921275 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":1,"thread_start_us":111,"threads_started":1}
I20260806 17:02:54.922051 17707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b444dd52de5d48a19249796aab3e1347" candidate_uuid: "ab8197e88b3a413ab3841456ccd766b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15c4a303d934243b47be0a30e0cd2d5" is_pre_election: true
I20260806 17:02:54.922163 17848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b444dd52de5d48a19249796aab3e1347" candidate_uuid: "ab8197e88b3a413ab3841456ccd766b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e9aff505ca4498a8128f450b898170" is_pre_election: true
I20260806 17:02:54.922226 17707 raft_consensus.cc:2454] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab8197e88b3a413ab3841456ccd766b3 in term 0.
I20260806 17:02:54.922410 17848 raft_consensus.cc:2454] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab8197e88b3a413ab3841456ccd766b3 in term 0.
I20260806 17:02:54.922614 17496 leader_election.cc:302] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab8197e88b3a413ab3841456ccd766b3, b15c4a303d934243b47be0a30e0cd2d5; no voters:
I20260806 17:02:54.922894 17930 raft_consensus.cc:2781] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:02:54.922962 17930 raft_consensus.cc:482] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:02:54.922991 17930 raft_consensus.cc:3037] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:54.924033 17930 raft_consensus.cc:504] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:54.924414 17930 leader_election.cc:288] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [CANDIDATE]: Term 1 election: Requested vote from peers b15c4a303d934243b47be0a30e0cd2d5 (127.9.111.66:43187), 45e9aff505ca4498a8128f450b898170 (127.9.111.67:42695)
I20260806 17:02:54.924787 17848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b444dd52de5d48a19249796aab3e1347" candidate_uuid: "ab8197e88b3a413ab3841456ccd766b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e9aff505ca4498a8128f450b898170"
I20260806 17:02:54.924806 17707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b444dd52de5d48a19249796aab3e1347" candidate_uuid: "ab8197e88b3a413ab3841456ccd766b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15c4a303d934243b47be0a30e0cd2d5"
I20260806 17:02:54.924928 17848 raft_consensus.cc:3037] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:54.924928 17707 raft_consensus.cc:3037] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:54.926152 17848 raft_consensus.cc:2454] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab8197e88b3a413ab3841456ccd766b3 in term 1.
I20260806 17:02:54.926152 17707 raft_consensus.cc:2454] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab8197e88b3a413ab3841456ccd766b3 in term 1.
I20260806 17:02:54.926446 17496 leader_election.cc:302] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab8197e88b3a413ab3841456ccd766b3, b15c4a303d934243b47be0a30e0cd2d5; no voters:
I20260806 17:02:54.926637 17930 raft_consensus.cc:2781] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:54.926874 17930 raft_consensus.cc:686] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [term 1 LEADER]: Becoming Leader. State: Replica: ab8197e88b3a413ab3841456ccd766b3, State: Running, Role: LEADER
I20260806 17:02:54.927161 17930 consensus_queue.cc:237] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:54.929423 17403 catalog_manager.cc:5495] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab8197e88b3a413ab3841456ccd766b3 (127.9.111.65). New cstate: current_term: 1 leader_uuid: "ab8197e88b3a413ab3841456ccd766b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } health_report { overall_health: UNKNOWN } } }
Aug 06 17:02:55 dist-test-slave-69rg krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035773, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.126@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM
I20260806 17:02:55.027985 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:55.019583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:40637 (local address 127.0.0.1:33872)
0806 17:02:55.019938 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:55.019952 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:55.020054 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:55.020324 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:55.020327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:55.020709 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:55.020805 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:55.020811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.021529 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.021531 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:55.022186 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:55.022198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.022637 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.022640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:55.022703 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:55.023317 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:55.023329 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:55.024736 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:55.026570 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:55.026578 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:55.026584 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:55.026821 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:55.027063 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:55.027064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:55.027066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:55.027142 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:55.027484 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:55.027489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:55.027590 (+ 101us) client_negotiation.cc:769] Sending connection context
0806 17:02:55.027680 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 17:02:55.027800 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":121,"threads_started":1}
I20260806 17:02:55.027998 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:55.019638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33872 (local address 127.9.111.126:40637)
0806 17:02:55.019885 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:55.019893 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:55.020021 (+ 128us) server_negotiation.cc:407] Connection header received
0806 17:02:55.020137 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:55.020139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:55.020180 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:55.020264 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:55.020910 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.021437 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.022310 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.022532 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.022612 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:55.024900 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:55.024904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:55.024907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:55.024934 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:55.026479 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:55.026907 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:55.026909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:55.026911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:55.026985 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:55.027215 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:55.027217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:55.027218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:55.027387 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:55.027461 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:55.027737 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 17:02:55.027865 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":163,"thread_start_us":82,"threads_started":1}
I20260806 17:02:55.031944 17403 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:33872:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 17:02:55.032591 17403 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:02:55.037408 17546 tablet_service.cc:1467] Processing CreateTablet for tablet 38943724feb54792a21f15d32c7da059 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cba577937f134293a230adc7f59e62e9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 17:02:55.037480 17828 tablet_service.cc:1467] Processing CreateTablet for tablet 38943724feb54792a21f15d32c7da059 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cba577937f134293a230adc7f59e62e9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 17:02:55.037443 17687 tablet_service.cc:1467] Processing CreateTablet for tablet 38943724feb54792a21f15d32c7da059 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cba577937f134293a230adc7f59e62e9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 17:02:55.037856 17828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38943724feb54792a21f15d32c7da059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:55.037856 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38943724feb54792a21f15d32c7da059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:55.037905 17687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38943724feb54792a21f15d32c7da059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:55.040412 17924 tablet_bootstrap.cc:492] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5: Bootstrap starting.
I20260806 17:02:55.040904 17925 tablet_bootstrap.cc:492] T 38943724feb54792a21f15d32c7da059 P 45e9aff505ca4498a8128f450b898170: Bootstrap starting.
I20260806 17:02:55.041036 17923 tablet_bootstrap.cc:492] T 38943724feb54792a21f15d32c7da059 P ab8197e88b3a413ab3841456ccd766b3: Bootstrap starting.
I20260806 17:02:55.041388 17924 tablet_bootstrap.cc:654] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:55.041788 17925 tablet_bootstrap.cc:654] T 38943724feb54792a21f15d32c7da059 P 45e9aff505ca4498a8128f450b898170: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:55.041953 17923 tablet_bootstrap.cc:654] T 38943724feb54792a21f15d32c7da059 P ab8197e88b3a413ab3841456ccd766b3: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:55.042712 17924 tablet_bootstrap.cc:492] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5: No bootstrap required, opened a new log
I20260806 17:02:55.042809 17924 ts_tablet_manager.cc:1397] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260806 17:02:55.042984 17925 tablet_bootstrap.cc:492] T 38943724feb54792a21f15d32c7da059 P 45e9aff505ca4498a8128f450b898170: No bootstrap required, opened a new log
I20260806 17:02:55.043057 17925 ts_tablet_manager.cc:1397] T 38943724feb54792a21f15d32c7da059 P 45e9aff505ca4498a8128f450b898170: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 17:02:55.043207 17923 tablet_bootstrap.cc:492] T 38943724feb54792a21f15d32c7da059 P ab8197e88b3a413ab3841456ccd766b3: No bootstrap required, opened a new log
I20260806 17:02:55.043275 17923 ts_tablet_manager.cc:1397] T 38943724feb54792a21f15d32c7da059 P ab8197e88b3a413ab3841456ccd766b3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 17:02:55.043264 17924 raft_consensus.cc:348] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.043354 17924 raft_consensus.cc:374] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:55.043376 17924 raft_consensus.cc:729] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15c4a303d934243b47be0a30e0cd2d5, State: Initialized, Role: FOLLOWER
I20260806 17:02:55.043465 17925 raft_consensus.cc:348] T 38943724feb54792a21f15d32c7da059 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.043509 17924 consensus_queue.cc:260] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.043567 17925 raft_consensus.cc:374] T 38943724feb54792a21f15d32c7da059 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:55.043601 17925 raft_consensus.cc:729] T 38943724feb54792a21f15d32c7da059 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e9aff505ca4498a8128f450b898170, State: Initialized, Role: FOLLOWER
I20260806 17:02:55.043726 17925 consensus_queue.cc:260] T 38943724feb54792a21f15d32c7da059 P 45e9aff505ca4498a8128f450b898170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.043704 17923 raft_consensus.cc:348] T 38943724feb54792a21f15d32c7da059 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.043758 17931 tablet_replica.cc:439] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } } }
I20260806 17:02:55.043824 17923 raft_consensus.cc:374] T 38943724feb54792a21f15d32c7da059 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:55.043862 17923 raft_consensus.cc:729] T 38943724feb54792a21f15d32c7da059 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab8197e88b3a413ab3841456ccd766b3, State: Initialized, Role: FOLLOWER
I20260806 17:02:55.043889 17931 tablet_replica.cc:442] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:55.044008 17925 ts_tablet_manager.cc:1428] T 38943724feb54792a21f15d32c7da059 P 45e9aff505ca4498a8128f450b898170: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 17:02:55.043977 17923 consensus_queue.cc:260] T 38943724feb54792a21f15d32c7da059 P ab8197e88b3a413ab3841456ccd766b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.044130 17929 tablet_replica.cc:439] T 38943724feb54792a21f15d32c7da059 P 45e9aff505ca4498a8128f450b898170: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } } }
I20260806 17:02:55.044250 17923 ts_tablet_manager.cc:1428] T 38943724feb54792a21f15d32c7da059 P ab8197e88b3a413ab3841456ccd766b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 17:02:55.044278 17929 tablet_replica.cc:442] T 38943724feb54792a21f15d32c7da059 P 45e9aff505ca4498a8128f450b898170: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:55.044838 17924 ts_tablet_manager.cc:1428] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260806 17:02:55.045171 17930 tablet_replica.cc:439] T 38943724feb54792a21f15d32c7da059 P ab8197e88b3a413ab3841456ccd766b3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } } }
I20260806 17:02:55.045320 17930 tablet_replica.cc:442] T 38943724feb54792a21f15d32c7da059 P ab8197e88b3a413ab3841456ccd766b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:55.239415 17931 raft_consensus.cc:482] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:55.239565 17931 raft_consensus.cc:504] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.240720 17931 leader_election.cc:288] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab8197e88b3a413ab3841456ccd766b3 (127.9.111.65:39521), 45e9aff505ca4498a8128f450b898170 (127.9.111.67:42695)
I20260806 17:02:55.245149 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:55.240999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:39521 (local address 127.9.111.66:39589)
0806 17:02:55.241327 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:55.241342 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:55.241460 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:55.241930 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:55.241933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:55.241966 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:55.242105 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:55.242110 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.242927 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.242930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:55.243892 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:55.243906 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.244604 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.244607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:55.244714 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:55.244799 (+ 85us) client_negotiation.cc:769] Sending connection context
0806 17:02:55.244942 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 17:02:55.245016 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":110,"thread_start_us":91,"threads_started":1}
I20260806 17:02:55.245148 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:55.241105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:47359 (local address 127.9.111.67:42695)
0806 17:02:55.241253 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:55.241257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:55.241589 (+ 332us) server_negotiation.cc:407] Connection header received
0806 17:02:55.241724 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:55.241726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:55.241768 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:55.241877 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:55.242213 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.242750 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.243957 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.244443 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.244578 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:55.244635 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:55.244921 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 17:02:55.245017 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 17:02:55.245189 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:55.241365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:39589 (local address 127.9.111.65:39521)
0806 17:02:55.241628 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:55.241633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:55.241650 (+ 17us) server_negotiation.cc:407] Connection header received
0806 17:02:55.241712 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:55.241715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:55.241766 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:55.241904 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:02:55.242247 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.242763 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.244039 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.244473 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.244623 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:55.244680 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:55.244965 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 17:02:55.245049 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":71,"threads_started":1}
I20260806 17:02:55.245633 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:55.240999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:42695 (local address 127.9.111.66:47359)
0806 17:02:55.241432 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:55.241443 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:55.241596 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:55.241902 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:55.241905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:55.241923 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:02:55.242089 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:55.242094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.242867 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.242870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:55.243825 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:55.243835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.244608 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.244620 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:55.244746 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:55.244766 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 17:02:55.245436 (+ 670us) client_negotiation.cc:240] Negotiation successful
0806 17:02:55.245522 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":84,"spinlock_wait_cycles":12032,"thread_start_us":131,"threads_started":1}
I20260806 17:02:55.246076 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38943724feb54792a21f15d32c7da059" candidate_uuid: "b15c4a303d934243b47be0a30e0cd2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab8197e88b3a413ab3841456ccd766b3" is_pre_election: true
I20260806 17:02:55.246068 17848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38943724feb54792a21f15d32c7da059" candidate_uuid: "b15c4a303d934243b47be0a30e0cd2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e9aff505ca4498a8128f450b898170" is_pre_election: true
I20260806 17:02:55.246217 17566 raft_consensus.cc:2454] T 38943724feb54792a21f15d32c7da059 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b15c4a303d934243b47be0a30e0cd2d5 in term 0.
I20260806 17:02:55.246248 17848 raft_consensus.cc:2454] T 38943724feb54792a21f15d32c7da059 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b15c4a303d934243b47be0a30e0cd2d5 in term 0.
I20260806 17:02:55.246642 17638 leader_election.cc:302] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 45e9aff505ca4498a8128f450b898170, ab8197e88b3a413ab3841456ccd766b3, b15c4a303d934243b47be0a30e0cd2d5; no voters:
I20260806 17:02:55.246941 17931 raft_consensus.cc:2781] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:02:55.247006 17931 raft_consensus.cc:482] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:02:55.247044 17931 raft_consensus.cc:3037] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:55.248222 17931 raft_consensus.cc:504] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.248636 17931 leader_election.cc:288] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 [CANDIDATE]: Term 1 election: Requested vote from peers ab8197e88b3a413ab3841456ccd766b3 (127.9.111.65:39521), 45e9aff505ca4498a8128f450b898170 (127.9.111.67:42695)
I20260806 17:02:55.249012 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38943724feb54792a21f15d32c7da059" candidate_uuid: "b15c4a303d934243b47be0a30e0cd2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab8197e88b3a413ab3841456ccd766b3"
I20260806 17:02:55.249034 17848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38943724feb54792a21f15d32c7da059" candidate_uuid: "b15c4a303d934243b47be0a30e0cd2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e9aff505ca4498a8128f450b898170"
I20260806 17:02:55.249177 17848 raft_consensus.cc:3037] T 38943724feb54792a21f15d32c7da059 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:55.249163 17566 raft_consensus.cc:3037] T 38943724feb54792a21f15d32c7da059 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:55.250268 17848 raft_consensus.cc:2454] T 38943724feb54792a21f15d32c7da059 P 45e9aff505ca4498a8128f450b898170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b15c4a303d934243b47be0a30e0cd2d5 in term 1.
I20260806 17:02:55.250280 17566 raft_consensus.cc:2454] T 38943724feb54792a21f15d32c7da059 P ab8197e88b3a413ab3841456ccd766b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b15c4a303d934243b47be0a30e0cd2d5 in term 1.
I20260806 17:02:55.250581 17638 leader_election.cc:302] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab8197e88b3a413ab3841456ccd766b3, b15c4a303d934243b47be0a30e0cd2d5; no voters:
I20260806 17:02:55.250797 17931 raft_consensus.cc:2781] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:55.251030 17931 raft_consensus.cc:686] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 [term 1 LEADER]: Becoming Leader. State: Replica: b15c4a303d934243b47be0a30e0cd2d5, State: Running, Role: LEADER
I20260806 17:02:55.251317 17931 consensus_queue.cc:237] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.252512 17970 tablet_replica.cc:439] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5: TxnStatusTablet state changed. Reason: New leader b15c4a303d934243b47be0a30e0cd2d5. Latest consensus state: current_term: 1 leader_uuid: "b15c4a303d934243b47be0a30e0cd2d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } } }
I20260806 17:02:55.252704 17970 tablet_replica.cc:442] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5: This TxnStatusTablet replica's current role is: LEADER
I20260806 17:02:55.252981 17972 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 17:02:55.253141 17972 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 17:02:55.253916 17403 catalog_manager.cc:5495] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b15c4a303d934243b47be0a30e0cd2d5 (127.9.111.66). New cstate: current_term: 1 leader_uuid: "b15c4a303d934243b47be0a30e0cd2d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } health_report { overall_health: UNKNOWN } } }
I20260806 17:02:55.287647 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:55.283559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.9.111.66:43187)
0806 17:02:55.283695 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:55.283699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:55.283715 (+ 16us) server_negotiation.cc:407] Connection header received
0806 17:02:55.283834 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:55.283838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:55.283905 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:55.284017 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:55.284247 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.285039 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.285896 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.286260 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.286360 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:55.286642 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:55.287231 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:55.287341 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:55.287432 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 17:02:55.287490 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 17:02:55.287642 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:55.283441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:43187 (local address 127.0.0.1:42780)
0806 17:02:55.283582 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:55.283600 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:55.283699 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:55.284032 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:55.284034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:55.284040 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 17:02:55.284136 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:55.284140 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.285148 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.285151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:55.285747 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:55.285754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.286380 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.286383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:55.286457 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:55.286504 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 17:02:55.287332 (+ 828us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 17:02:55.287346 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 17:02:55.287397 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 17:02:55.287532 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 17:02:55.291308 17566 raft_consensus.cc:1261] T 38943724feb54792a21f15d32c7da059 P ab8197e88b3a413ab3841456ccd766b3 [term 1 FOLLOWER]: Refusing update from remote peer b15c4a303d934243b47be0a30e0cd2d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:55.291371 17848 raft_consensus.cc:1261] T 38943724feb54792a21f15d32c7da059 P 45e9aff505ca4498a8128f450b898170 [term 1 FOLLOWER]: Refusing update from remote peer b15c4a303d934243b47be0a30e0cd2d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:55.291854 17931 consensus_queue.cc:1035] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:55.292026 17970 consensus_queue.cc:1035] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:55.293872 17929 tablet_replica.cc:439] T 38943724feb54792a21f15d32c7da059 P 45e9aff505ca4498a8128f450b898170: TxnStatusTablet state changed. Reason: New leader b15c4a303d934243b47be0a30e0cd2d5. Latest consensus state: current_term: 1 leader_uuid: "b15c4a303d934243b47be0a30e0cd2d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } } }
I20260806 17:02:55.294000 17929 tablet_replica.cc:442] T 38943724feb54792a21f15d32c7da059 P 45e9aff505ca4498a8128f450b898170: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:55.293741 17930 tablet_replica.cc:439] T 38943724feb54792a21f15d32c7da059 P ab8197e88b3a413ab3841456ccd766b3: TxnStatusTablet state changed. Reason: New leader b15c4a303d934243b47be0a30e0cd2d5. Latest consensus state: current_term: 1 leader_uuid: "b15c4a303d934243b47be0a30e0cd2d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } } }
I20260806 17:02:55.295092 17930 tablet_replica.cc:442] T 38943724feb54792a21f15d32c7da059 P ab8197e88b3a413ab3841456ccd766b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:55.296605 17970 tablet_replica.cc:439] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b15c4a303d934243b47be0a30e0cd2d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } } }
I20260806 17:02:55.296732 17970 tablet_replica.cc:442] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5: This TxnStatusTablet replica's current role is: LEADER
I20260806 17:02:55.296983 17929 tablet_replica.cc:439] T 38943724feb54792a21f15d32c7da059 P 45e9aff505ca4498a8128f450b898170: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b15c4a303d934243b47be0a30e0cd2d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } } }
I20260806 17:02:55.297096 17929 tablet_replica.cc:442] T 38943724feb54792a21f15d32c7da059 P 45e9aff505ca4498a8128f450b898170: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:55.297086 17970 tablet_replica.cc:439] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b15c4a303d934243b47be0a30e0cd2d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } } }
I20260806 17:02:55.297190 17970 tablet_replica.cc:442] T 38943724feb54792a21f15d32c7da059 P b15c4a303d934243b47be0a30e0cd2d5: This TxnStatusTablet replica's current role is: LEADER
I20260806 17:02:55.299561 17930 tablet_replica.cc:439] T 38943724feb54792a21f15d32c7da059 P ab8197e88b3a413ab3841456ccd766b3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b15c4a303d934243b47be0a30e0cd2d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } } }
I20260806 17:02:55.299710 17930 tablet_replica.cc:442] T 38943724feb54792a21f15d32c7da059 P ab8197e88b3a413ab3841456ccd766b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 17:02:55.307413 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:55.303438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56328 (local address 127.9.111.65:39521)
0806 17:02:55.303566 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:55.303570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:55.303777 (+ 207us) server_negotiation.cc:407] Connection header received
0806 17:02:55.303917 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:55.303919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:55.303973 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:55.304058 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:55.304336 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.304846 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.306083 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.306302 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.306379 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:55.306647 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:55.307085 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:55.307168 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:55.307265 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 17:02:55.307311 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 17:02:55.319207 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:55.314874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:38637 (local address 127.9.111.66:43187)
0806 17:02:55.315003 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:55.315007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:55.315146 (+ 139us) server_negotiation.cc:407] Connection header received
0806 17:02:55.315301 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:55.315305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:55.315358 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:55.315436 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:55.315842 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.316627 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.317937 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.318307 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.318411 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:55.318742 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:55.318826 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:55.318899 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:55.319009 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 17:02:55.319066 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 17:02:55.319238 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:55.314751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:43187 (local address 127.9.111.65:38637)
0806 17:02:55.315052 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:55.315068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:55.315166 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:55.315561 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:55.315564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:55.315573 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 17:02:55.315719 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:55.315725 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.316754 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.316757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:55.317753 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:55.317764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.318434 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.318438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:55.318539 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:55.318602 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 17:02:55.318913 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 17:02:55.318919 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 17:02:55.318981 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 17:02:55.319066 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":128,"threads_started":1}
I20260806 17:02:55.329046 17848 raft_consensus.cc:1261] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170 [term 1 FOLLOWER]: Refusing update from remote peer ab8197e88b3a413ab3841456ccd766b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:55.329071 17707 raft_consensus.cc:1261] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5 [term 1 FOLLOWER]: Refusing update from remote peer ab8197e88b3a413ab3841456ccd766b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:55.329692 17930 consensus_queue.cc:1035] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:55.330454 17954 consensus_queue.cc:1035] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:55.342013 17978 mvcc.cc:205] Tried to move back new op lower bound from 7315602535743176704 to 7315602534104461312. Current Snapshot: MvccSnapshot[applied={T|T < 7315602535743176704}]
I20260806 17:02:55.358533 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:55.354631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:50417 (local address 127.9.111.65:39521)
0806 17:02:55.354833 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:55.354842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:55.354954 (+ 112us) server_negotiation.cc:407] Connection header received
0806 17:02:55.355159 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:55.355162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:55.355227 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:55.355308 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:55.355537 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.356339 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.357180 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.357529 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.357625 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:55.358008 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:55.358116 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:55.358195 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:55.358345 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 17:02:55.358397 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 17:02:55.358570 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:55.354537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:39521 (local address 127.9.111.66:50417)
0806 17:02:55.354874 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:55.354890 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:55.355031 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:55.355317 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:55.355319 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:55.355327 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 17:02:55.355436 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:55.355443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.356453 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.356456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:55.357071 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:55.357080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.357653 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.357658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:55.357749 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:55.357864 (+ 115us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 17:02:55.358221 (+ 357us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 17:02:55.358228 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 17:02:55.358308 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 17:02:55.358391 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":125,"threads_started":1}
I20260806 17:02:55.383211 17403 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:55.383416 17403 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:55.385921 17403 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02: Sending DeleteTablet for 3 replicas of tablet b444dd52de5d48a19249796aab3e1347
I20260806 17:02:55.386525 17546 tablet_service.cc:1514] Processing DeleteTablet for tablet b444dd52de5d48a19249796aab3e1347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:55 UTC) from {username='oryx', principal='oryx/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:56312
I20260806 17:02:55.386677 17828 tablet_service.cc:1514] Processing DeleteTablet for tablet b444dd52de5d48a19249796aab3e1347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:55 UTC) from {username='oryx', principal='oryx/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:40972
I20260806 17:02:55.387037 17999 tablet_replica.cc:330] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170: stopping tablet replica
I20260806 17:02:55.387172 17999 raft_consensus.cc:2229] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:55.387315 17999 raft_consensus.cc:2258] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:55.387511 17687 tablet_service.cc:1514] Processing DeleteTablet for tablet b444dd52de5d48a19249796aab3e1347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:55 UTC) from {username='oryx', principal='oryx/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:42768
I20260806 17:02:55.388309 17998 tablet_replica.cc:330] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3: stopping tablet replica
I20260806 17:02:55.388486 17998 raft_consensus.cc:2229] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:02:55.388947 17998 raft_consensus.cc:2258] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:55.389712 17998 ts_tablet_manager.cc:1905] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:55.392350 17999 ts_tablet_manager.cc:1905] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:55.393047 17998 ts_tablet_manager.cc:1918] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 17:02:55.393209 17998 log.cc:1196] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/wal/wals/b444dd52de5d48a19249796aab3e1347
I20260806 17:02:55.393625 17998 ts_tablet_manager.cc:1939] T b444dd52de5d48a19249796aab3e1347 P ab8197e88b3a413ab3841456ccd766b3: Deleting consensus metadata
I20260806 17:02:55.394575 17389 catalog_manager.cc:4841] TS ab8197e88b3a413ab3841456ccd766b3 (127.9.111.65:39521): tablet b444dd52de5d48a19249796aab3e1347 (table test-table [id=b94d8aca8abb4a34b287b914113ab4dc]) successfully deleted
I20260806 17:02:55.395363 17999 ts_tablet_manager.cc:1918] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 17:02:55.395454 17999 log.cc:1196] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/b444dd52de5d48a19249796aab3e1347
I20260806 17:02:55.395712 17999 ts_tablet_manager.cc:1939] T b444dd52de5d48a19249796aab3e1347 P 45e9aff505ca4498a8128f450b898170: Deleting consensus metadata
I20260806 17:02:55.396512 17391 catalog_manager.cc:4841] TS 45e9aff505ca4498a8128f450b898170 (127.9.111.67:42695): tablet b444dd52de5d48a19249796aab3e1347 (table test-table [id=b94d8aca8abb4a34b287b914113ab4dc]) successfully deleted
I20260806 17:02:55.398061 18000 tablet_replica.cc:330] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5: stopping tablet replica
I20260806 17:02:55.398303 18000 raft_consensus.cc:2229] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:55.398510 18000 raft_consensus.cc:2258] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:55.399507 18000 ts_tablet_manager.cc:1905] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:55.403215 18000 ts_tablet_manager.cc:1918] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 17:02:55.403393 18000 log.cc:1196] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/b444dd52de5d48a19249796aab3e1347
I20260806 17:02:55.403767 18000 ts_tablet_manager.cc:1939] T b444dd52de5d48a19249796aab3e1347 P b15c4a303d934243b47be0a30e0cd2d5: Deleting consensus metadata
I20260806 17:02:55.404657 17390 catalog_manager.cc:4841] TS b15c4a303d934243b47be0a30e0cd2d5 (127.9.111.66:43187): tablet b444dd52de5d48a19249796aab3e1347 (table test-table [id=b94d8aca8abb4a34b287b914113ab4dc]) successfully deleted
Aug 06 17:02:55 dist-test-slave-69rg krb5kdc[17358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 17:02:55 dist-test-slave-69rg krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:02:55 dist-test-slave-69rg krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
W20260806 17:02:55.434170 17963 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:02:55.420350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33886 (local address 127.9.111.126:40637)
0806 17:02:55.420503 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:55.420508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:55.420777 (+ 269us) server_negotiation.cc:407] Connection header received
0806 17:02:55.420925 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:55.420928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:55.420986 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:55.421082 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:55.422320 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.423251 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.424532 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.424922 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.425036 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:55.434023 (+ 8987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33886: BlockingRecv error: recv got EOF from 127.0.0.1:33886 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 17:02:55 dist-test-slave-69rg krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM
I20260806 17:02:55.445405 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:55.435785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33898 (local address 127.9.111.126:40637)
0806 17:02:55.435908 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:55.435912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:55.436109 (+ 197us) server_negotiation.cc:407] Connection header received
0806 17:02:55.436236 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:55.436239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:55.436298 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:55.436386 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:55.437110 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.437996 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.438989 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.439342 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.439447 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:55.442023 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:55.442027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:55.442030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:55.442060 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:55.443521 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:55.443965 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:55.443968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:55.443969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:55.444150 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:55.444437 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:55.444442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:55.444444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:55.444709 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:55.444860 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:55.445094 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 17:02:55.445224 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 17:02:55.448311 17403 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:02:55.448782 17403 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:02:55.453871 17546 tablet_service.cc:1467] Processing CreateTablet for tablet 2e53d1e3bb0a406cbe831aa3f960cf48 (DEFAULT_TABLE table=test-table [id=dec13efcfa9b443d85991e3dc57f9ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:55.453984 17687 tablet_service.cc:1467] Processing CreateTablet for tablet 2e53d1e3bb0a406cbe831aa3f960cf48 (DEFAULT_TABLE table=test-table [id=dec13efcfa9b443d85991e3dc57f9ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:55.454196 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e53d1e3bb0a406cbe831aa3f960cf48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:55.454299 17687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e53d1e3bb0a406cbe831aa3f960cf48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:55.456291 17924 tablet_bootstrap.cc:492] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5: Bootstrap starting.
I20260806 17:02:55.456835 17828 tablet_service.cc:1467] Processing CreateTablet for tablet 2e53d1e3bb0a406cbe831aa3f960cf48 (DEFAULT_TABLE table=test-table [id=dec13efcfa9b443d85991e3dc57f9ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:55.457130 17828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e53d1e3bb0a406cbe831aa3f960cf48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:55.457238 17924 tablet_bootstrap.cc:654] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:55.458541 17924 tablet_bootstrap.cc:492] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5: No bootstrap required, opened a new log
I20260806 17:02:55.458653 17924 ts_tablet_manager.cc:1397] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260806 17:02:55.459049 17924 raft_consensus.cc:348] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.459172 17924 raft_consensus.cc:374] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:55.459211 17924 raft_consensus.cc:729] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15c4a303d934243b47be0a30e0cd2d5, State: Initialized, Role: FOLLOWER
I20260806 17:02:55.459332 17924 consensus_queue.cc:260] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.459656 17924 ts_tablet_manager.cc:1428] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260806 17:02:55.459676 17923 tablet_bootstrap.cc:492] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3: Bootstrap starting.
I20260806 17:02:55.460675 17923 tablet_bootstrap.cc:654] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:55.461352 17925 tablet_bootstrap.cc:492] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170: Bootstrap starting.
I20260806 17:02:55.462061 17923 tablet_bootstrap.cc:492] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3: No bootstrap required, opened a new log
I20260806 17:02:55.462170 17923 ts_tablet_manager.cc:1397] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 17:02:55.462234 17925 tablet_bootstrap.cc:654] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:55.462698 17923 raft_consensus.cc:348] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.462831 17923 raft_consensus.cc:374] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:55.462874 17923 raft_consensus.cc:729] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab8197e88b3a413ab3841456ccd766b3, State: Initialized, Role: FOLLOWER
I20260806 17:02:55.463135 17923 consensus_queue.cc:260] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.463493 17923 ts_tablet_manager.cc:1428] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 17:02:55.464887 17925 tablet_bootstrap.cc:492] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170: No bootstrap required, opened a new log
I20260806 17:02:55.464982 17925 ts_tablet_manager.cc:1397] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 17:02:55.465389 17925 raft_consensus.cc:348] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.465515 17925 raft_consensus.cc:374] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:55.465548 17925 raft_consensus.cc:729] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e9aff505ca4498a8128f450b898170, State: Initialized, Role: FOLLOWER
I20260806 17:02:55.465687 17925 consensus_queue.cc:260] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.465904 17982 raft_consensus.cc:482] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:55.465998 17982 raft_consensus.cc:504] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.466380 17982 leader_election.cc:288] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab8197e88b3a413ab3841456ccd766b3 (127.9.111.65:39521), 45e9aff505ca4498a8128f450b898170 (127.9.111.67:42695)
I20260806 17:02:55.466723 17848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e53d1e3bb0a406cbe831aa3f960cf48" candidate_uuid: "b15c4a303d934243b47be0a30e0cd2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e9aff505ca4498a8128f450b898170" is_pre_election: true
I20260806 17:02:55.466857 17848 raft_consensus.cc:2454] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b15c4a303d934243b47be0a30e0cd2d5 in term 0.
I20260806 17:02:55.466845 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e53d1e3bb0a406cbe831aa3f960cf48" candidate_uuid: "b15c4a303d934243b47be0a30e0cd2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab8197e88b3a413ab3841456ccd766b3" is_pre_election: true
I20260806 17:02:55.466964 17566 raft_consensus.cc:2454] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b15c4a303d934243b47be0a30e0cd2d5 in term 0.
I20260806 17:02:55.467154 17925 ts_tablet_manager.cc:1428] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 17:02:55.467187 17640 leader_election.cc:302] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45e9aff505ca4498a8128f450b898170, b15c4a303d934243b47be0a30e0cd2d5; no voters:
I20260806 17:02:55.467558 17982 raft_consensus.cc:2781] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:02:55.467618 17982 raft_consensus.cc:482] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:02:55.467655 17982 raft_consensus.cc:3037] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:55.468825 17982 raft_consensus.cc:504] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.469166 17982 leader_election.cc:288] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [CANDIDATE]: Term 1 election: Requested vote from peers ab8197e88b3a413ab3841456ccd766b3 (127.9.111.65:39521), 45e9aff505ca4498a8128f450b898170 (127.9.111.67:42695)
I20260806 17:02:55.469475 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e53d1e3bb0a406cbe831aa3f960cf48" candidate_uuid: "b15c4a303d934243b47be0a30e0cd2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab8197e88b3a413ab3841456ccd766b3"
I20260806 17:02:55.469583 17566 raft_consensus.cc:3037] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:55.469585 17848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e53d1e3bb0a406cbe831aa3f960cf48" candidate_uuid: "b15c4a303d934243b47be0a30e0cd2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e9aff505ca4498a8128f450b898170"
I20260806 17:02:55.469682 17848 raft_consensus.cc:3037] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:55.470738 17566 raft_consensus.cc:2454] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b15c4a303d934243b47be0a30e0cd2d5 in term 1.
I20260806 17:02:55.470808 17848 raft_consensus.cc:2454] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b15c4a303d934243b47be0a30e0cd2d5 in term 1.
I20260806 17:02:55.471082 17638 leader_election.cc:302] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 45e9aff505ca4498a8128f450b898170, ab8197e88b3a413ab3841456ccd766b3, b15c4a303d934243b47be0a30e0cd2d5; no voters:
I20260806 17:02:55.471226 17982 raft_consensus.cc:2781] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:55.471329 17982 raft_consensus.cc:686] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [term 1 LEADER]: Becoming Leader. State: Replica: b15c4a303d934243b47be0a30e0cd2d5, State: Running, Role: LEADER
I20260806 17:02:55.471473 17982 consensus_queue.cc:237] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.473066 17404 catalog_manager.cc:5495] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b15c4a303d934243b47be0a30e0cd2d5 (127.9.111.66). New cstate: current_term: 1 leader_uuid: "b15c4a303d934243b47be0a30e0cd2d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } health_report { overall_health: UNKNOWN } } }
I20260806 17:02:55.486200 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:55.483059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42784 (local address 127.9.111.66:43187)
0806 17:02:55.483196 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:55.483201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:55.483218 (+ 17us) server_negotiation.cc:407] Connection header received
0806 17:02:55.483274 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:55.483277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:55.483332 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:55.483411 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:55.483664 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.484205 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.485074 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.485309 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.485422 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:55.485682 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:55.485763 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:55.485904 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:55.485996 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 17:02:55.486053 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 17:02:55.488720 17848 raft_consensus.cc:1261] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170 [term 1 FOLLOWER]: Refusing update from remote peer b15c4a303d934243b47be0a30e0cd2d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:55.488735 17566 raft_consensus.cc:1261] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3 [term 1 FOLLOWER]: Refusing update from remote peer b15c4a303d934243b47be0a30e0cd2d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:55.489240 17982 consensus_queue.cc:1035] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:55.489423 17931 consensus_queue.cc:1035] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:55.499620 17403 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:55.499794 17403 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:55.502014 17403 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02: Sending DeleteTablet for 3 replicas of tablet 2e53d1e3bb0a406cbe831aa3f960cf48
I20260806 17:02:55.502602 17687 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e53d1e3bb0a406cbe831aa3f960cf48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:55 UTC) from {username='oryx', principal='oryx/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:42768
I20260806 17:02:55.502671 17546 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e53d1e3bb0a406cbe831aa3f960cf48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:55 UTC) from {username='oryx', principal='oryx/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:56312
I20260806 17:02:55.502779 18000 tablet_replica.cc:330] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5: stopping tablet replica
I20260806 17:02:55.502923 18000 raft_consensus.cc:2229] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:02:55.503176 17828 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e53d1e3bb0a406cbe831aa3f960cf48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:55 UTC) from {username='oryx', principal='oryx/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:40972
I20260806 17:02:55.503219 18000 raft_consensus.cc:2258] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:55.503369 17999 tablet_replica.cc:330] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170: stopping tablet replica
I20260806 17:02:55.503386 17998 tablet_replica.cc:330] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3: stopping tablet replica
I20260806 17:02:55.503487 17998 raft_consensus.cc:2229] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:55.503484 17999 raft_consensus.cc:2229] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:55.503602 17998 raft_consensus.cc:2258] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:55.503609 17999 raft_consensus.cc:2258] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:55.504565 17403 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 17:02:55.504699 18000 ts_tablet_manager.cc:1905] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 17:02:55.504994 17403 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:02:55.506261 17999 ts_tablet_manager.cc:1905] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:55.507421 18000 ts_tablet_manager.cc:1918] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:02:55.507504 18000 log.cc:1196] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/2e53d1e3bb0a406cbe831aa3f960cf48
I20260806 17:02:55.507774 18000 ts_tablet_manager.cc:1939] T 2e53d1e3bb0a406cbe831aa3f960cf48 P b15c4a303d934243b47be0a30e0cd2d5: Deleting consensus metadata
I20260806 17:02:55.508801 17999 ts_tablet_manager.cc:1918] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:02:55.508885 17999 log.cc:1196] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/2e53d1e3bb0a406cbe831aa3f960cf48
I20260806 17:02:55.509141 17999 ts_tablet_manager.cc:1939] T 2e53d1e3bb0a406cbe831aa3f960cf48 P 45e9aff505ca4498a8128f450b898170: Deleting consensus metadata
I20260806 17:02:55.509898 17391 catalog_manager.cc:4841] TS 45e9aff505ca4498a8128f450b898170 (127.9.111.67:42695): tablet 2e53d1e3bb0a406cbe831aa3f960cf48 (table test-table [id=dec13efcfa9b443d85991e3dc57f9ad4]) successfully deleted
I20260806 17:02:55.510879 17390 catalog_manager.cc:4841] TS b15c4a303d934243b47be0a30e0cd2d5 (127.9.111.66:43187): tablet 2e53d1e3bb0a406cbe831aa3f960cf48 (table test-table [id=dec13efcfa9b443d85991e3dc57f9ad4]) successfully deleted
I20260806 17:02:55.513546 17546 tablet_service.cc:1467] Processing CreateTablet for tablet 17a053228ad24d31b972f180039f9454 (DEFAULT_TABLE table=test-table [id=6d205a92eb2541ceaf55b43859e5b4a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:55.513886 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a053228ad24d31b972f180039f9454. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:55.514456 17998 ts_tablet_manager.cc:1905] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:55.516661 17998 ts_tablet_manager.cc:1918] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:02:55.516728 17998 log.cc:1196] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-0/wal/wals/2e53d1e3bb0a406cbe831aa3f960cf48
I20260806 17:02:55.516728 17828 tablet_service.cc:1467] Processing CreateTablet for tablet 17a053228ad24d31b972f180039f9454 (DEFAULT_TABLE table=test-table [id=6d205a92eb2541ceaf55b43859e5b4a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:55.516991 17828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a053228ad24d31b972f180039f9454. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:55.517000 17998 ts_tablet_manager.cc:1939] T 2e53d1e3bb0a406cbe831aa3f960cf48 P ab8197e88b3a413ab3841456ccd766b3: Deleting consensus metadata
I20260806 17:02:55.519271 17925 tablet_bootstrap.cc:492] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170: Bootstrap starting.
I20260806 17:02:55.520061 17925 tablet_bootstrap.cc:654] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:55.521107 17389 catalog_manager.cc:4841] TS ab8197e88b3a413ab3841456ccd766b3 (127.9.111.65:39521): tablet 2e53d1e3bb0a406cbe831aa3f960cf48 (table test-table [id=dec13efcfa9b443d85991e3dc57f9ad4]) successfully deleted
I20260806 17:02:55.521395 17923 tablet_bootstrap.cc:492] T 17a053228ad24d31b972f180039f9454 P ab8197e88b3a413ab3841456ccd766b3: Bootstrap starting.
I20260806 17:02:55.522295 17923 tablet_bootstrap.cc:654] T 17a053228ad24d31b972f180039f9454 P ab8197e88b3a413ab3841456ccd766b3: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:55.523837 17925 tablet_bootstrap.cc:492] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170: No bootstrap required, opened a new log
I20260806 17:02:55.523928 17925 ts_tablet_manager.cc:1397] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260806 17:02:55.524475 17925 raft_consensus.cc:348] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.524614 17925 raft_consensus.cc:374] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:55.524654 17925 raft_consensus.cc:729] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e9aff505ca4498a8128f450b898170, State: Initialized, Role: FOLLOWER
I20260806 17:02:55.524788 17925 consensus_queue.cc:260] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.525035 17925 ts_tablet_manager.cc:1428] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 17:02:55.525066 17687 tablet_service.cc:1467] Processing CreateTablet for tablet 17a053228ad24d31b972f180039f9454 (DEFAULT_TABLE table=test-table [id=6d205a92eb2541ceaf55b43859e5b4a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:02:55.525359 17687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a053228ad24d31b972f180039f9454. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:55.527678 17924 tablet_bootstrap.cc:492] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5: Bootstrap starting.
I20260806 17:02:55.528517 17924 tablet_bootstrap.cc:654] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:55.533739 17924 tablet_bootstrap.cc:492] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5: No bootstrap required, opened a new log
I20260806 17:02:55.533872 17924 ts_tablet_manager.cc:1397] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260806 17:02:55.534322 17924 raft_consensus.cc:348] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.534444 17924 raft_consensus.cc:374] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:55.534482 17924 raft_consensus.cc:729] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15c4a303d934243b47be0a30e0cd2d5, State: Initialized, Role: FOLLOWER
I20260806 17:02:55.534610 17924 consensus_queue.cc:260] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.534701 17923 tablet_bootstrap.cc:492] T 17a053228ad24d31b972f180039f9454 P ab8197e88b3a413ab3841456ccd766b3: No bootstrap required, opened a new log
I20260806 17:02:55.534825 17923 ts_tablet_manager.cc:1397] T 17a053228ad24d31b972f180039f9454 P ab8197e88b3a413ab3841456ccd766b3: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20260806 17:02:55.534873 17924 ts_tablet_manager.cc:1428] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 17:02:55.535320 17923 raft_consensus.cc:348] T 17a053228ad24d31b972f180039f9454 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.535444 17923 raft_consensus.cc:374] T 17a053228ad24d31b972f180039f9454 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:55.535480 17923 raft_consensus.cc:729] T 17a053228ad24d31b972f180039f9454 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab8197e88b3a413ab3841456ccd766b3, State: Initialized, Role: FOLLOWER
I20260806 17:02:55.535601 17923 consensus_queue.cc:260] T 17a053228ad24d31b972f180039f9454 P ab8197e88b3a413ab3841456ccd766b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.535866 17923 ts_tablet_manager.cc:1428] T 17a053228ad24d31b972f180039f9454 P ab8197e88b3a413ab3841456ccd766b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 17:02:55.665696 17931 raft_consensus.cc:482] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:02:55.665918 17931 raft_consensus.cc:504] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.666429 17931 leader_election.cc:288] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab8197e88b3a413ab3841456ccd766b3 (127.9.111.65:39521), 45e9aff505ca4498a8128f450b898170 (127.9.111.67:42695)
I20260806 17:02:55.666934 17848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a053228ad24d31b972f180039f9454" candidate_uuid: "b15c4a303d934243b47be0a30e0cd2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e9aff505ca4498a8128f450b898170" is_pre_election: true
I20260806 17:02:55.666934 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a053228ad24d31b972f180039f9454" candidate_uuid: "b15c4a303d934243b47be0a30e0cd2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab8197e88b3a413ab3841456ccd766b3" is_pre_election: true
I20260806 17:02:55.667102 17848 raft_consensus.cc:2454] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b15c4a303d934243b47be0a30e0cd2d5 in term 0.
I20260806 17:02:55.667102 17566 raft_consensus.cc:2454] T 17a053228ad24d31b972f180039f9454 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b15c4a303d934243b47be0a30e0cd2d5 in term 0.
I20260806 17:02:55.667435 17640 leader_election.cc:302] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45e9aff505ca4498a8128f450b898170, b15c4a303d934243b47be0a30e0cd2d5; no voters:
I20260806 17:02:55.667609 17931 raft_consensus.cc:2781] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:02:55.667675 17931 raft_consensus.cc:482] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:02:55.667759 17931 raft_consensus.cc:3037] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:55.668772 17931 raft_consensus.cc:504] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.669160 17931 leader_election.cc:288] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [CANDIDATE]: Term 1 election: Requested vote from peers ab8197e88b3a413ab3841456ccd766b3 (127.9.111.65:39521), 45e9aff505ca4498a8128f450b898170 (127.9.111.67:42695)
I20260806 17:02:55.669543 17848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a053228ad24d31b972f180039f9454" candidate_uuid: "b15c4a303d934243b47be0a30e0cd2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e9aff505ca4498a8128f450b898170"
I20260806 17:02:55.669574 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a053228ad24d31b972f180039f9454" candidate_uuid: "b15c4a303d934243b47be0a30e0cd2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab8197e88b3a413ab3841456ccd766b3"
I20260806 17:02:55.669694 17848 raft_consensus.cc:3037] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:55.669720 17566 raft_consensus.cc:3037] T 17a053228ad24d31b972f180039f9454 P ab8197e88b3a413ab3841456ccd766b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:55.670665 17848 raft_consensus.cc:2454] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b15c4a303d934243b47be0a30e0cd2d5 in term 1.
I20260806 17:02:55.670981 17566 raft_consensus.cc:2454] T 17a053228ad24d31b972f180039f9454 P ab8197e88b3a413ab3841456ccd766b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b15c4a303d934243b47be0a30e0cd2d5 in term 1.
I20260806 17:02:55.671069 17640 leader_election.cc:302] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45e9aff505ca4498a8128f450b898170, b15c4a303d934243b47be0a30e0cd2d5; no voters:
I20260806 17:02:55.671304 17931 raft_consensus.cc:2781] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:55.671401 17931 raft_consensus.cc:686] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [term 1 LEADER]: Becoming Leader. State: Replica: b15c4a303d934243b47be0a30e0cd2d5, State: Running, Role: LEADER
I20260806 17:02:55.671550 17931 consensus_queue.cc:237] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } }
I20260806 17:02:55.673416 17403 catalog_manager.cc:5495] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b15c4a303d934243b47be0a30e0cd2d5 (127.9.111.66). New cstate: current_term: 1 leader_uuid: "b15c4a303d934243b47be0a30e0cd2d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b15c4a303d934243b47be0a30e0cd2d5" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 43187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 } health_report { overall_health: UNKNOWN } } }
I20260806 17:02:55.699057 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:55.695922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:43187 (local address 127.9.111.66:60147)
0806 17:02:55.696073 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:55.696092 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:55.696181 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:55.696429 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:55.696431 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:55.696437 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 17:02:55.696575 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:55.696580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.697279 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.697284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:55.697919 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:55.697927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.698367 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.698370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:55.698489 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:55.698519 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 17:02:55.698787 (+ 268us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 17:02:55.698794 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 17:02:55.698854 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 17:02:55.698936 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":41}
I20260806 17:02:55.699079 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:55.696043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:60147 (local address 127.9.111.66:43187)
0806 17:02:55.696139 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:55.696144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:55.696156 (+ 12us) server_negotiation.cc:407] Connection header received
0806 17:02:55.696289 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:55.696293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:55.696336 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:55.696422 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:02:55.696674 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.697172 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.698032 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:55.698261 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:55.698369 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:55.698628 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:02:55.698700 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:02:55.698774 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:55.698862 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 17:02:55.698909 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":29}
I20260806 17:02:55.705068 17566 raft_consensus.cc:1261] T 17a053228ad24d31b972f180039f9454 P ab8197e88b3a413ab3841456ccd766b3 [term 1 FOLLOWER]: Refusing update from remote peer b15c4a303d934243b47be0a30e0cd2d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:55.705039 17848 raft_consensus.cc:1261] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170 [term 1 FOLLOWER]: Refusing update from remote peer b15c4a303d934243b47be0a30e0cd2d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:02:55.705446 17982 consensus_queue.cc:1035] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab8197e88b3a413ab3841456ccd766b3" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:55.705637 17931 consensus_queue.cc:1035] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45e9aff505ca4498a8128f450b898170" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 42695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:02:55.736531 17403 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:55.736677 17403 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:02:55.738592 17403 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cca8e0737954706b30ebace8f6c5c02: Sending DeleteTablet for 3 replicas of tablet 17a053228ad24d31b972f180039f9454
I20260806 17:02:55.739116 17546 tablet_service.cc:1514] Processing DeleteTablet for tablet 17a053228ad24d31b972f180039f9454 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:55 UTC) from {username='oryx', principal='oryx/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:56312
I20260806 17:02:55.739171 17686 tablet_service.cc:1514] Processing DeleteTablet for tablet 17a053228ad24d31b972f180039f9454 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:55 UTC) from {username='oryx', principal='oryx/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:42768
I20260806 17:02:55.739197 17828 tablet_service.cc:1514] Processing DeleteTablet for tablet 17a053228ad24d31b972f180039f9454 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:02:55 UTC) from {username='oryx', principal='oryx/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:40972
I20260806 17:02:55.739367 17999 tablet_replica.cc:330] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170: stopping tablet replica
I20260806 17:02:55.739364 17998 tablet_replica.cc:330] T 17a053228ad24d31b972f180039f9454 P ab8197e88b3a413ab3841456ccd766b3: stopping tablet replica
I20260806 17:02:55.739500 17998 raft_consensus.cc:2229] T 17a053228ad24d31b972f180039f9454 P ab8197e88b3a413ab3841456ccd766b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:55.739501 17999 raft_consensus.cc:2229] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:02:55.739636 17998 raft_consensus.cc:2258] T 17a053228ad24d31b972f180039f9454 P ab8197e88b3a413ab3841456ccd766b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:55.739639 17999 raft_consensus.cc:2258] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:55.739837 18000 tablet_replica.cc:330] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5: stopping tablet replica
I20260806 17:02:55.739981 18000 raft_consensus.cc:2229] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:02:55.740180 18000 raft_consensus.cc:2258] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:02:55.740460 17998 ts_tablet_manager.cc:1905] T 17a053228ad24d31b972f180039f9454 P ab8197e88b3a413ab3841456ccd766b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:55.740531 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 17478
I20260806 17:02:55.740790 17999 ts_tablet_manager.cc:1905] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:55.742362 18000 ts_tablet_manager.cc:1905] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:02:55.743556 17999 ts_tablet_manager.cc:1918] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 17:02:55.743621 17999 log.cc:1196] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/17a053228ad24d31b972f180039f9454
I20260806 17:02:55.743899 17999 ts_tablet_manager.cc:1939] T 17a053228ad24d31b972f180039f9454 P 45e9aff505ca4498a8128f450b898170: Deleting consensus metadata
I20260806 17:02:55.744580 17391 catalog_manager.cc:4841] TS 45e9aff505ca4498a8128f450b898170 (127.9.111.67:42695): tablet 17a053228ad24d31b972f180039f9454 (table test-table [id=6d205a92eb2541ceaf55b43859e5b4a9]) successfully deleted
I20260806 17:02:55.744618 18000 ts_tablet_manager.cc:1918] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 17:02:55.744683 18000 log.cc:1196] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/17a053228ad24d31b972f180039f9454
I20260806 17:02:55.745033 18000 ts_tablet_manager.cc:1939] T 17a053228ad24d31b972f180039f9454 P b15c4a303d934243b47be0a30e0cd2d5: Deleting consensus metadata
I20260806 17:02:55.745702 17390 catalog_manager.cc:4841] TS b15c4a303d934243b47be0a30e0cd2d5 (127.9.111.66:43187): tablet 17a053228ad24d31b972f180039f9454 (table test-table [id=6d205a92eb2541ceaf55b43859e5b4a9]) successfully deleted
W20260806 17:02:55.751156 17389 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.111.65:39521 (error 108)
I20260806 17:02:55.751245 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 17621
I20260806 17:02:55.753584 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:55.752096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:39521 (local address 127.0.0.1:56344)
0806 17:02:55.753426 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:55.753496 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.65:39521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1239,"thread_start_us":81,"threads_started":1}
W20260806 17:02:55.753758 17389 catalog_manager.cc:4584] TS ab8197e88b3a413ab3841456ccd766b3 (127.9.111.65:39521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 17a053228ad24d31b972f180039f9454: Network error: Client connection negotiation failed: client connection to 127.9.111.65:39521: connect: Connection refused (error 111)
I20260806 17:02:55.758980 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 17763
I20260806 17:02:55.766000 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 17371
2026-08-06T17:02:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3643 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:02:55 dist-test-slave-69rg krb5kdc[18031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:02:55 dist-test-slave-69rg krb5kdc[18031](info): set up 2 sockets
Aug 06 17:02:55 dist-test-slave-69rg krb5kdc[18031](info): commencing operation
krb5kdc: starting...
W20260806 17:02:56.830042 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:02:56 dist-test-slave-69rg krb5kdc[18031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:02:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:02:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:02:56.963148 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:41027
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:41027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:57.077677 18044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:57.078090 18044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:57.078210 18044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:57.082419 18044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:57.082528 18044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:57.082568 18044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:57.082608 18044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:57.082641 18044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:02:57.088171 18044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:41027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:41027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:57.089529 18044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:57.090764 18044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 17:02:57.100852 18044 server_base.cc:1034] running on GCE node
W20260806 17:02:57.100780 18049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:57.100793 18052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:57.100795 18050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:57.101723 18044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:57.103050 18044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:57.104290 18044 hybrid_clock.cc:648] HybridClock initialized: now 1786035777104259 us; error 52 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T17:02:57Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1344 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:02:57 dist-test-slave-69rg krb5kdc[18057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:02:57 dist-test-slave-69rg krb5kdc[18057](info): set up 2 sockets
Aug 06 17:02:57 dist-test-slave-69rg krb5kdc[18057](info): commencing operation
krb5kdc: starting...
W20260806 17:02:58.177537 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:02:58 dist-test-slave-69rg krb5kdc[18057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:02:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:02:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:02:58.314347 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:38999
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:38999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:58.429549 18071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:58.429910 18071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:58.430027 18071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:58.434109 18071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:58.434195 18071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:58.434234 18071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:58.434288 18071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:58.434327 18071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:02:58.439924 18071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:38999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:38999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:58.441115 18071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:58.442301 18071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:58.451050 18079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:58.451043 18077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:58.451136 18076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:58.451325 18071 server_base.cc:1034] running on GCE node
I20260806 17:02:58.452155 18071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:58.453366 18071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:58.454564 18071 hybrid_clock.cc:648] HybridClock initialized: now 1786035778454543 us; error 35 us; skew 500 ppm
Aug 06 17:02:58 dist-test-slave-69rg krb5kdc[18057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035778, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:58.457505 18071 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:02:58.458596 18071 webserver.cc:460] Webserver started at http://127.9.111.126:44651/ using document root <none> and password file <none>
I20260806 17:02:58.459228 18071 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:58.459328 18071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:58.459576 18071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:58.461746 18071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "73efe58efc174015a51a2ea702fb9ade"
format_stamp: "Formatted at 2026-08-06 17:02:58 on dist-test-slave-69rg"
server_key: "c74e5aefdcf2ebe68aaf0e59ba0ff23a"
server_key_iv: "f7fcba7de49a6b82b4031477bc931d5e"
server_key_version: "encryptionkey@0"
I20260806 17:02:58.462421 18071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "73efe58efc174015a51a2ea702fb9ade"
format_stamp: "Formatted at 2026-08-06 17:02:58 on dist-test-slave-69rg"
server_key: "c74e5aefdcf2ebe68aaf0e59ba0ff23a"
server_key_iv: "f7fcba7de49a6b82b4031477bc931d5e"
server_key_version: "encryptionkey@0"
I20260806 17:02:58.466382 18071 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:02:58.468858 18086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:58.470395 18071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:58.470574 18071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "73efe58efc174015a51a2ea702fb9ade"
format_stamp: "Formatted at 2026-08-06 17:02:58 on dist-test-slave-69rg"
server_key: "c74e5aefdcf2ebe68aaf0e59ba0ff23a"
server_key_iv: "f7fcba7de49a6b82b4031477bc931d5e"
server_key_version: "encryptionkey@0"
I20260806 17:02:58.470717 18071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:58.493571 18071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:58.496992 18071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:58.497265 18071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:58.506090 18138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:38999 every 8 connection(s)
I20260806 17:02:58.506095 18071 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:38999
I20260806 17:02:58.507242 18071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:02:58.510129 18139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:02:58.512184 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 18071
I20260806 17:02:58.512341 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:02:58.512643 9661 external_mini_cluster.cc:1428] Setting key ed6470c5f6d8c1cca08524739025d810
I20260806 17:02:58.516782 18139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade: Bootstrap starting.
I20260806 17:02:58.519477 18139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade: Neither blocks nor log segments found. Creating new log.
I20260806 17:02:58.520363 18139 log.cc:824] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade: Log is configured to *not* fsync() on all Append() calls
Aug 06 17:02:58 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:58.522531 18139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade: No bootstrap required, opened a new log
I20260806 17:02:58.526386 18139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73efe58efc174015a51a2ea702fb9ade" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38999 } }
I20260806 17:02:58.526665 18139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:02:58.526726 18139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73efe58efc174015a51a2ea702fb9ade, State: Initialized, Role: FOLLOWER
I20260806 17:02:58.527218 18139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73efe58efc174015a51a2ea702fb9ade" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38999 } }
I20260806 17:02:58.527397 18139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:02:58.527498 18139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:02:58.527621 18139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:02:58.528028 18142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:58.514203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.9.111.126:38999)
0806 17:02:58.514690 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:58.514711 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:58.514757 (+ 46us) server_negotiation.cc:407] Connection header received
0806 17:02:58.515596 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:58.515620 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:58.516078 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:58.516431 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:58.517223 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:58.518724 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:58.519571 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:58.520255 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:58.520417 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:58.523065 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:58.523077 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:58.523090 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:58.523118 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:58.524830 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:58.525290 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:58.525296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:58.525302 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:58.525424 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:58.525718 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:58.525721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:58.525723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:58.526203 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:58.526425 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:58.526729 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 17:02:58.527093 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":135,"threads_started":1}
I20260806 17:02:58.528718 18139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73efe58efc174015a51a2ea702fb9ade" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38999 } }
I20260806 17:02:58.529090 18139 leader_election.cc:302] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73efe58efc174015a51a2ea702fb9ade; no voters:
I20260806 17:02:58.529440 18139 leader_election.cc:288] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:02:58.529552 18144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:02:58.529755 18144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [term 1 LEADER]: Becoming Leader. State: Replica: 73efe58efc174015a51a2ea702fb9ade, State: Running, Role: LEADER
I20260806 17:02:58.530083 18144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73efe58efc174015a51a2ea702fb9ade" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38999 } }
I20260806 17:02:58.530809 18139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:58.531311 18144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73efe58efc174015a51a2ea702fb9ade. Latest consensus state: current_term: 1 leader_uuid: "73efe58efc174015a51a2ea702fb9ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73efe58efc174015a51a2ea702fb9ade" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38999 } } }
I20260806 17:02:58.531432 18144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:58.531804 18149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:58.532019 18146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73efe58efc174015a51a2ea702fb9ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73efe58efc174015a51a2ea702fb9ade" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38999 } } }
I20260806 17:02:58.533353 18146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:58.535575 18149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:02:58.541873 18149 catalog_manager.cc:1347] Generated new cluster ID: 61f7403d4a404f13a0d72ee842893335
I20260806 17:02:58.541950 18149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:58.556398 18149 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:02:58.557396 18149 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:58.573302 18149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade: Generated new TSK 0
I20260806 17:02:58.574141 18149 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
I20260806 17:02:58.612337 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38999
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:58.727321 18166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:58.727628 18166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:58.727675 18166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:58.731922 18166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:58.731997 18166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:58.732059 18166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:02:58.737843 18166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:58.738963 18166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:58.739964 18166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 17:02:58.750092 18166 server_base.cc:1034] running on GCE node
W20260806 17:02:58.750105 18175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:58.750025 18172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:58.750008 18173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:58.750698 18166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:58.751405 18166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:58.752612 18166 hybrid_clock.cc:648] HybridClock initialized: now 1786035778752596 us; error 27 us; skew 500 ppm
Aug 06 17:02:58 dist-test-slave-69rg krb5kdc[18057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035778, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:58.755599 18166 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:02:58.756511 18166 webserver.cc:460] Webserver started at http://127.9.111.65:44611/ using document root <none> and password file <none>
I20260806 17:02:58.757103 18166 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:58.757198 18166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:58.757447 18166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:58.759673 18166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "3edf0ac264544acf84cfbbd9b2e9a0be"
format_stamp: "Formatted at 2026-08-06 17:02:58 on dist-test-slave-69rg"
server_key: "5d3b2872a38fc3f08085be612c675252"
server_key_iv: "3fe8054a1d43663ec6817d6488799f44"
server_key_version: "encryptionkey@0"
I20260806 17:02:58.760280 18166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "3edf0ac264544acf84cfbbd9b2e9a0be"
format_stamp: "Formatted at 2026-08-06 17:02:58 on dist-test-slave-69rg"
server_key: "5d3b2872a38fc3f08085be612c675252"
server_key_iv: "3fe8054a1d43663ec6817d6488799f44"
server_key_version: "encryptionkey@0"
I20260806 17:02:58.764448 18166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260806 17:02:58.767112 18182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:58.768379 18166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 17:02:58.768545 18166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "3edf0ac264544acf84cfbbd9b2e9a0be"
format_stamp: "Formatted at 2026-08-06 17:02:58 on dist-test-slave-69rg"
server_key: "5d3b2872a38fc3f08085be612c675252"
server_key_iv: "3fe8054a1d43663ec6817d6488799f44"
server_key_version: "encryptionkey@0"
I20260806 17:02:58.768682 18166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:58.793092 18166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:58.796833 18166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:58.797065 18166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:58.797734 18166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:58.798728 18166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:58.798810 18166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:58.798884 18166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:58.798934 18166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:58.808441 18166 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:44469
I20260806 17:02:58.808461 18297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:44469 every 8 connection(s)
I20260806 17:02:58.809453 18166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:02:58.810384 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 18166
I20260806 17:02:58.810539 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:02:58.810850 9661 external_mini_cluster.cc:1428] Setting key 7711025889a5e9daaaaf944b064d7878
Aug 06 17:02:58 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035778, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.111.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.66@KRBTEST.COM" created.
I20260806 17:02:58.827628 18142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:58.812531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:41897 (local address 127.9.111.126:38999)
0806 17:02:58.812659 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:58.812663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:58.814036 (+ 1373us) server_negotiation.cc:407] Connection header received
0806 17:02:58.815176 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:58.815178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:58.815229 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:58.815321 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:58.817176 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:58.817725 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:58.819821 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:58.820061 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:58.820158 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:58.823370 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:58.823373 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:58.823375 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:58.823407 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:58.824982 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:58.825606 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:58.825615 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:58.825616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:58.825700 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:58.826174 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:58.826177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:58.826181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:58.826473 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:58.826594 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:58.827185 (+ 591us) server_negotiation.cc:299] Negotiation successful
0806 17:02:58.827364 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 17:02:58.828323 18300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:58.813088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.65:41897)
0806 17:02:58.813885 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:58.813938 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:58.814910 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:58.815569 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:58.815581 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:58.816183 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:58.816979 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:58.817005 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:58.817946 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:58.817953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:58.819657 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:58.819678 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:58.820250 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:58.820256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:58.820499 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:58.821187 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:58.821213 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:58.823121 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:58.825142 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:58.825150 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:58.825169 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:58.825474 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:58.825806 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:58.825809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:58.825811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:58.826054 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:58.826605 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:58.826611 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:58.826901 (+ 290us) client_negotiation.cc:769] Sending connection context
0806 17:02:58.827076 (+ 175us) client_negotiation.cc:240] Negotiation successful
0806 17:02:58.827370 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":168,"threads_started":1}
I20260806 17:02:58.829640 18298 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38999
I20260806 17:02:58.829944 18298 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:58.830595 18298 heartbeater.cc:511] Master 127.9.111.126:38999 requested a full tablet report, sending...
I20260806 17:02:58.832437 18103 ts_manager.cc:194] Registered new tserver with Master: 3edf0ac264544acf84cfbbd9b2e9a0be (127.9.111.65:44469)
I20260806 17:02:58.833712 18103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:41897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
I20260806 17:02:58.846079 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38999
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:58.964023 18303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:58.964323 18303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:58.964370 18303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:58.968677 18303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:58.968758 18303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:58.968822 18303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:02:58.975354 18303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:58.976658 18303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:58.977797 18303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:58.988404 18311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:58.988443 18308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:58.988489 18303 server_base.cc:1034] running on GCE node
W20260806 17:02:58.988443 18309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:58.989120 18303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:58.989888 18303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:58.991086 18303 hybrid_clock.cc:648] HybridClock initialized: now 1786035778991069 us; error 28 us; skew 500 ppm
Aug 06 17:02:58 dist-test-slave-69rg krb5kdc[18057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035778, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:58.994427 18303 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:02:58.995522 18303 webserver.cc:460] Webserver started at http://127.9.111.66:33365/ using document root <none> and password file <none>
I20260806 17:02:58.996205 18303 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:58.996313 18303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:58.996580 18303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:58.998996 18303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "b861217973bc417fa67cfd428bb345c5"
format_stamp: "Formatted at 2026-08-06 17:02:58 on dist-test-slave-69rg"
server_key: "e2ff4f1f0dda795d578999f8a30938d4"
server_key_iv: "11a35072422f97e43dcc58725cca385b"
server_key_version: "encryptionkey@0"
I20260806 17:02:58.999656 18303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "b861217973bc417fa67cfd428bb345c5"
format_stamp: "Formatted at 2026-08-06 17:02:58 on dist-test-slave-69rg"
server_key: "e2ff4f1f0dda795d578999f8a30938d4"
server_key_iv: "11a35072422f97e43dcc58725cca385b"
server_key_version: "encryptionkey@0"
I20260806 17:02:59.004176 18303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 17:02:59.007193 18319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:59.008729 18303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:59.008901 18303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "b861217973bc417fa67cfd428bb345c5"
format_stamp: "Formatted at 2026-08-06 17:02:58 on dist-test-slave-69rg"
server_key: "e2ff4f1f0dda795d578999f8a30938d4"
server_key_iv: "11a35072422f97e43dcc58725cca385b"
server_key_version: "encryptionkey@0"
I20260806 17:02:59.009047 18303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:59.032577 18303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:59.036259 18303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:59.036501 18303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:59.037174 18303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:59.038218 18303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:59.038272 18303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:59.038347 18303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:59.038394 18303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:59.048965 18303 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:41133
I20260806 17:02:59.048995 18436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:41133 every 8 connection(s)
I20260806 17:02:59.050104 18303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:02:59.054487 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 18303
I20260806 17:02:59.054598 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:02:59.054878 9661 external_mini_cluster.cc:1428] Setting key c8d5653527f053777da3b3d2892312fe
Aug 06 17:02:59 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035778, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.111.67@KRBTEST.COM; defaulting to no policy
I20260806 17:02:59.066944 18142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:59.052871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:57511 (local address 127.9.111.126:38999)
0806 17:02:59.053096 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:59.053101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:59.053879 (+ 778us) server_negotiation.cc:407] Connection header received
0806 17:02:59.054855 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:59.054858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:59.054922 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:59.055002 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:59.056507 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.057265 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.058684 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.058920 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.059061 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:59.062448 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:59.062451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:59.062454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:59.062483 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:59.064071 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:59.064863 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:59.064866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:59.064867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:59.064959 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:59.065494 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:59.065498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:59.065499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:59.065720 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:59.065809 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:59.066656 (+ 847us) server_negotiation.cc:299] Negotiation successful
0806 17:02:59.066798 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260806 17:02:59.067878 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:59.052729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.66:57511)
0806 17:02:59.053666 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:59.053723 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:59.054610 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:59.055315 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:59.055324 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:59.055748 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:59.056339 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:59.056361 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.057425 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.057428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:59.058543 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:59.058553 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.059110 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.059122 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:59.059334 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:59.060048 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:59.060074 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:59.062176 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:59.064218 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:59.064229 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:59.064253 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:59.064704 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:59.065066 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:59.065071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:59.065077 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:59.065320 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:59.065875 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:59.065883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:59.066271 (+ 388us) client_negotiation.cc:769] Sending connection context
0806 17:02:59.066543 (+ 272us) client_negotiation.cc:240] Negotiation successful
0806 17:02:59.066886 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":181,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.67@KRBTEST.COM" created.
I20260806 17:02:59.069450 18437 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38999
I20260806 17:02:59.069729 18437 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:59.070331 18437 heartbeater.cc:511] Master 127.9.111.126:38999 requested a full tablet report, sending...
I20260806 17:02:59.071599 18103 ts_manager.cc:194] Registered new tserver with Master: b861217973bc417fa67cfd428bb345c5 (127.9.111.66:41133)
I20260806 17:02:59.072392 18103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:57511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
I20260806 17:02:59.090708 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38999
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:59.205382 18443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:59.205674 18443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:59.205720 18443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:59.209918 18443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:59.209992 18443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:59.210095 18443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:02:59.215827 18443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:59.216982 18443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:59.218226 18443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:59.228528 18449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:59.228528 18448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:59.228636 18451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:59.228842 18443 server_base.cc:1034] running on GCE node
I20260806 17:02:59.229342 18443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:59.230154 18443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:59.231383 18443 hybrid_clock.cc:648] HybridClock initialized: now 1786035779231366 us; error 35 us; skew 500 ppm
Aug 06 17:02:59 dist-test-slave-69rg krb5kdc[18057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035779, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:59.234508 18443 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:02:59.235543 18443 webserver.cc:460] Webserver started at http://127.9.111.67:42251/ using document root <none> and password file <none>
I20260806 17:02:59.236137 18443 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:59.236236 18443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:59.236491 18443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:02:59.238736 18443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "a0d32cae4c5f41629336210f657f06d3"
format_stamp: "Formatted at 2026-08-06 17:02:59 on dist-test-slave-69rg"
server_key: "a9df30955923067cb9609cd9ca9556ce"
server_key_iv: "4310c873edc0c68fa3b6d58744b61803"
server_key_version: "encryptionkey@0"
I20260806 17:02:59.239341 18443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "a0d32cae4c5f41629336210f657f06d3"
format_stamp: "Formatted at 2026-08-06 17:02:59 on dist-test-slave-69rg"
server_key: "a9df30955923067cb9609cd9ca9556ce"
server_key_iv: "4310c873edc0c68fa3b6d58744b61803"
server_key_version: "encryptionkey@0"
I20260806 17:02:59.243265 18443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:02:59.245864 18460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:59.247118 18443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:02:59.247260 18443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "a0d32cae4c5f41629336210f657f06d3"
format_stamp: "Formatted at 2026-08-06 17:02:59 on dist-test-slave-69rg"
server_key: "a9df30955923067cb9609cd9ca9556ce"
server_key_iv: "4310c873edc0c68fa3b6d58744b61803"
server_key_version: "encryptionkey@0"
I20260806 17:02:59.247396 18443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:59.260411 18443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:59.263896 18443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:59.264117 18443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:59.264743 18443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:59.265623 18443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:59.265697 18443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:59.265767 18443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:59.265823 18443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:59.275573 18443 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:36581
I20260806 17:02:59.275590 18576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:36581 every 8 connection(s)
I20260806 17:02:59.276546 18443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:02:59.278510 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 18443
I20260806 17:02:59.278671 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:02:59.278956 9661 external_mini_cluster.cc:1428] Setting key 83f51abf73092c56934ab6f3e0bf7ce4
Aug 06 17:02:59 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035779, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:02:59.290262 18142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:59.278511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:52313 (local address 127.9.111.126:38999)
0806 17:02:59.278694 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:59.278699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:59.279797 (+ 1098us) server_negotiation.cc:407] Connection header received
0806 17:02:59.280750 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:59.280753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:59.280807 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:59.280907 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:59.282254 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.282779 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.283901 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.284109 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.284185 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:59.286729 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:02:59.286732 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:02:59.286734 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:02:59.286760 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:02:59.288202 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:59.288701 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:59.288705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:59.288706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:59.288797 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:02:59.289113 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:02:59.289117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:02:59.289119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:02:59.289327 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:02:59.289414 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 17:02:59.289979 (+ 565us) server_negotiation.cc:299] Negotiation successful
0806 17:02:59.290105 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 17:02:59.291147 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:59.278896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.67:52313)
0806 17:02:59.279643 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:59.279698 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:59.280558 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:59.281146 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:59.281153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:59.281524 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:59.282123 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:59.282138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.282902 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.282905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:59.283792 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:59.283801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.284244 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.284246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:59.284393 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:59.284964 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:59.284981 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:59.286561 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:02:59.288319 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:59.288324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:59.288336 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:59.288596 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:59.288885 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:02:59.288887 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:02:59.288888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:02:59.289021 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:02:59.289420 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:02:59.289424 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:02:59.289668 (+ 244us) client_negotiation.cc:769] Sending connection context
0806 17:02:59.289917 (+ 249us) client_negotiation.cc:240] Negotiation successful
0806 17:02:59.290198 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":137,"threads_started":1}
I20260806 17:02:59.292483 18577 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38999
I20260806 17:02:59.292771 18577 heartbeater.cc:464] Registering TS with master...
I20260806 17:02:59.293359 18577 heartbeater.cc:511] Master 127.9.111.126:38999 requested a full tablet report, sending...
I20260806 17:02:59.294471 18102 ts_manager.cc:194] Registered new tserver with Master: a0d32cae4c5f41629336210f657f06d3 (127.9.111.67:36581)
I20260806 17:02:59.295095 18102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:52313
I20260806 17:02:59.305210 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.65.keytab.
Entry for principal oryx/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.65.keytab.
WARNING: no policy specified for oryx/127.9.111.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.111.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.66.keytab.
Entry for principal oryx/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.66.keytab.
WARNING: no policy specified for oryx/127.9.111.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.111.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.67.keytab.
Entry for principal oryx/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.67.keytab.
I20260806 17:02:59.386549 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 18166
I20260806 17:02:59.393019 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 18303
I20260806 17:02:59.400122 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 18443
I20260806 17:02:59.406700 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 18071
I20260806 17:02:59.414426 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:38999
--webserver_interface=127.9.111.126
--webserver_port=44651
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:38999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:02:59.527387 18586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:59.527679 18586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:59.527721 18586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:59.531824 18586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:02:59.531889 18586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:59.531908 18586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:59.531925 18586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:02:59.531973 18586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:02:59.537465 18586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:38999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:38999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=44651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:59.538581 18586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:59.539664 18586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:59.548382 18591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:59.548404 18592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:59.548388 18594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:59.548686 18586 server_base.cc:1034] running on GCE node
I20260806 17:02:59.549357 18586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:59.550902 18586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:59.552127 18586 hybrid_clock.cc:648] HybridClock initialized: now 1786035779552080 us; error 52 us; skew 500 ppm
Aug 06 17:02:59 dist-test-slave-69rg krb5kdc[18057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035779, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:59.555378 18586 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:02:59.556353 18586 webserver.cc:460] Webserver started at http://127.9.111.126:44651/ using document root <none> and password file <none>
I20260806 17:02:59.556951 18586 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:59.557054 18586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:59.561045 18586 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 17:02:59.563278 18601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:59.564546 18586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 17:02:59.564694 18586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "73efe58efc174015a51a2ea702fb9ade"
format_stamp: "Formatted at 2026-08-06 17:02:58 on dist-test-slave-69rg"
server_key: "c74e5aefdcf2ebe68aaf0e59ba0ff23a"
server_key_iv: "f7fcba7de49a6b82b4031477bc931d5e"
server_key_version: "encryptionkey@0"
I20260806 17:02:59.565248 18586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:59.581624 18586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:59.584698 18586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:59.584928 18586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:59.592662 18586 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:38999
I20260806 17:02:59.592693 18655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:38999 every 8 connection(s)
I20260806 17:02:59.593762 18586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:02:59.598284 18656 sys_catalog.cc:263] Verifying existing consensus state
I20260806 17:02:59.600118 18656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade: Bootstrap starting.
I20260806 17:02:59.601884 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 18586
I20260806 17:02:59.602373 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:44469
--local_ip_for_outbound_sockets=127.9.111.65
--tserver_master_addrs=127.9.111.126:38999
--webserver_port=44611
--webserver_interface=127.9.111.65
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.111.65
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:02:59.622696 18656 log.cc:824] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade: Log is configured to *not* fsync() on all Append() calls
I20260806 17:02:59.631505 18656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 17:02:59.631974 18656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade: Bootstrap complete.
I20260806 17:02:59.636567 18656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73efe58efc174015a51a2ea702fb9ade" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38999 } }
I20260806 17:02:59.637036 18656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73efe58efc174015a51a2ea702fb9ade, State: Initialized, Role: FOLLOWER
I20260806 17:02:59.637614 18656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73efe58efc174015a51a2ea702fb9ade" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38999 } }
I20260806 17:02:59.637773 18656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:02:59.637889 18656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:02:59.638010 18656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [term 1 FOLLOWER]: Advancing to term 2
I20260806 17:02:59.640157 18656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73efe58efc174015a51a2ea702fb9ade" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38999 } }
I20260806 17:02:59.640527 18656 leader_election.cc:302] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73efe58efc174015a51a2ea702fb9ade; no voters:
I20260806 17:02:59.640903 18656 leader_election.cc:288] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 17:02:59.641011 18660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [term 2 FOLLOWER]: Leader election won for term 2
I20260806 17:02:59.641214 18660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [term 2 LEADER]: Becoming Leader. State: Replica: 73efe58efc174015a51a2ea702fb9ade, State: Running, Role: LEADER
I20260806 17:02:59.641539 18660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73efe58efc174015a51a2ea702fb9ade" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38999 } }
I20260806 17:02:59.642294 18656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:02:59.643014 18661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "73efe58efc174015a51a2ea702fb9ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73efe58efc174015a51a2ea702fb9ade" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38999 } } }
I20260806 17:02:59.643159 18661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:59.643517 18662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73efe58efc174015a51a2ea702fb9ade. Latest consensus state: current_term: 2 leader_uuid: "73efe58efc174015a51a2ea702fb9ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73efe58efc174015a51a2ea702fb9ade" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38999 } } }
I20260806 17:02:59.643612 18662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade [sys.catalog]: This master's current role is: LEADER
I20260806 17:02:59.644609 18666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:02:59.648110 18666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:02:59.648784 18666 catalog_manager.cc:1259] Loaded cluster ID: 61f7403d4a404f13a0d72ee842893335
I20260806 17:02:59.648869 18666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:02:59.651273 18666 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:02:59.651927 18666 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 73efe58efc174015a51a2ea702fb9ade: Loaded TSK: 0
I20260806 17:02:59.653123 18666 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 17:02:59.729327 18658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:59.729602 18658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:59.729650 18658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:59.735843 18658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:59.735929 18658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:59.735991 18658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:02:59.741500 18658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.65.keytab
--principal=oryx/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:44469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=44611
--tserver_master_addrs=127.9.111.126:38999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:59.742825 18658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:59.744046 18658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:59.755086 18682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:59.755432 18658 server_base.cc:1034] running on GCE node
W20260806 17:02:59.755112 18686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:59.755230 18683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:59.756048 18658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:59.756785 18658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:59.757982 18658 hybrid_clock.cc:648] HybridClock initialized: now 1786035779757951 us; error 49 us; skew 500 ppm
Aug 06 17:02:59 dist-test-slave-69rg krb5kdc[18057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:59.760946 18658 init.cc:377] Logged in from keytab as oryx/127.9.111.65@KRBTEST.COM (short username oryx)
I20260806 17:02:59.761987 18658 webserver.cc:460] Webserver started at http://127.9.111.65:44611/ using document root <none> and password file <none>
I20260806 17:02:59.762562 18658 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:59.762636 18658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:59.766718 18658 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 17:02:59.768944 18693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:59.770390 18658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:02:59.770510 18658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "3edf0ac264544acf84cfbbd9b2e9a0be"
format_stamp: "Formatted at 2026-08-06 17:02:58 on dist-test-slave-69rg"
server_key: "5d3b2872a38fc3f08085be612c675252"
server_key_iv: "3fe8054a1d43663ec6817d6488799f44"
server_key_version: "encryptionkey@0"
I20260806 17:02:59.771209 18658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:02:59.795691 18658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:02:59.799376 18658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:02:59.799540 18658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:02:59.800113 18658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:02:59.800951 18658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:02:59.801002 18658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:59.801043 18658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:02:59.801064 18658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:59.809672 18658 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:44469
I20260806 17:02:59.809764 18808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:44469 every 8 connection(s)
I20260806 17:02:59.810863 18658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:02:59.821054 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 18658
I20260806 17:02:59.821509 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:41133
--local_ip_for_outbound_sockets=127.9.111.66
--tserver_master_addrs=127.9.111.126:38999
--webserver_port=33365
--webserver_interface=127.9.111.66
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.111.66
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 17:02:59 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:02:59 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:02:59.824304 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:59.813087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.65:40029)
0806 17:02:59.813805 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:59.813891 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:59.814696 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:59.815869 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:59.815875 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:59.816250 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:59.816712 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:59.816725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.818024 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.818027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:59.818978 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:59.818992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.819768 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.819771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:59.819907 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:59.820513 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:59.820529 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:59.823520 (+ 2991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":143,"threads_started":1}
W20260806 17:02:59.824837 18809 heartbeater.cc:644] Failed to heartbeat to 127.9.111.126:38999 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.111.126:38999: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
W20260806 17:02:59.825332 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:02:59.812996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:40029 (local address 127.9.111.126:38999)
0806 17:02:59.813440 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:59.813457 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:59.814005 (+ 548us) server_negotiation.cc:407] Connection header received
0806 17:02:59.815125 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:59.815136 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:59.815442 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:59.815746 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:59.816908 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.817895 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.819145 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.819622 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.819747 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:59.823841 (+ 4094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.65:40029: BlockingRecv error: recv got EOF from 127.9.111.65:40029 (error 108)
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":120,"threads_started":1}
Aug 06 17:02:59 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:02:59 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:02:59.831672 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:59.825329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.65:58679)
0806 17:02:59.825677 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:59.825694 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:59.825782 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:59.826097 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:59.826099 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:59.826239 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:59.826338 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:59.826343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.827536 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.827539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:59.828104 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:59.828113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.828834 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.828837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:59.828907 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:59.829489 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:59.829502 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:59.831567 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273}
W20260806 17:02:59.831986 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:02:59.825412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:58679 (local address 127.9.111.126:38999)
0806 17:02:59.825671 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:59.825678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:59.825860 (+ 182us) server_negotiation.cc:407] Connection header received
0806 17:02:59.825913 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:59.825918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:59.825979 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:59.826072 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:59.826472 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.827383 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.828272 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.828698 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.828845 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:59.831798 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.65:58679: BlockingRecv error: recv got EOF from 127.9.111.65:58679 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":143,"threads_started":1}
Aug 06 17:02:59 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:02:59 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:02:59.838253 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:02:59.832480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.65:32875)
0806 17:02:59.832623 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 17:02:59.832638 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:02:59.832732 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:02:59.832982 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:02:59.832985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:02:59.833158 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:02:59.833230 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:59.833235 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.834027 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.834032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:59.834613 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:02:59.834621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.835077 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.835081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:02:59.835156 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:59.835700 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:02:59.835711 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:02:59.838084 (+ 2373us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 17:02:59.838356 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:02:59.832561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:32875 (local address 127.9.111.126:38999)
0806 17:02:59.832688 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 17:02:59.832692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:02:59.832705 (+ 13us) server_negotiation.cc:407] Connection header received
0806 17:02:59.832828 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:02:59.832830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:02:59.832880 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:02:59.832964 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:02:59.833347 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.833922 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.834746 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:02:59.834967 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:02:59.835075 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:02:59.838219 (+ 3144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.65:32875: BlockingRecv error: recv got EOF from 127.9.111.65:32875 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 17:02:59.838711 18809 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 17:02:59.942463 18813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:02:59.942795 18813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:02:59.942900 18813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:02:59.947350 18813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:02:59.947448 18813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:02:59.947543 18813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:02:59.953163 18813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.66.keytab
--principal=oryx/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:41133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=33365
--tserver_master_addrs=127.9.111.126:38999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:02:59.954385 18813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:02:59.955410 18813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:02:59.965039 18823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:59.965046 18821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:02:59.965080 18819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:02:59.965328 18813 server_base.cc:1034] running on GCE node
I20260806 17:02:59.965888 18813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:02:59.966554 18813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:02:59.967751 18813 hybrid_clock.cc:648] HybridClock initialized: now 1786035779967734 us; error 34 us; skew 500 ppm
Aug 06 17:02:59 dist-test-slave-69rg krb5kdc[18057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:02:59.970726 18813 init.cc:377] Logged in from keytab as oryx/127.9.111.66@KRBTEST.COM (short username oryx)
I20260806 17:02:59.971561 18813 webserver.cc:460] Webserver started at http://127.9.111.66:33365/ using document root <none> and password file <none>
I20260806 17:02:59.972078 18813 fs_manager.cc:359] Metadata directory not provided
I20260806 17:02:59.972151 18813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:02:59.976397 18813 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 17:02:59.978679 18830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:02:59.979944 18813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 17:02:59.980048 18813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "b861217973bc417fa67cfd428bb345c5"
format_stamp: "Formatted at 2026-08-06 17:02:58 on dist-test-slave-69rg"
server_key: "e2ff4f1f0dda795d578999f8a30938d4"
server_key_iv: "11a35072422f97e43dcc58725cca385b"
server_key_version: "encryptionkey@0"
I20260806 17:02:59.980592 18813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:03:00.001705 18813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:03:00.005146 18813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:03:00.005352 18813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:03:00.005975 18813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:03:00.006857 18813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:03:00.006906 18813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:00.006945 18813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:03:00.006961 18813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:00.018036 18813 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:41133
I20260806 17:03:00.018062 18945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:41133 every 8 connection(s)
I20260806 17:03:00.019351 18813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:03:00.020304 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 18813
I20260806 17:03:00.020862 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:36581
--local_ip_for_outbound_sockets=127.9.111.67
--tserver_master_addrs=127.9.111.126:38999
--webserver_port=42251
--webserver_interface=127.9.111.67
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.111.67
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 17:03:00 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:00 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
W20260806 17:03:00.033802 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:00.021515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:57341 (local address 127.9.111.126:38999)
0806 17:03:00.021694 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:00.021699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:00.023177 (+ 1478us) server_negotiation.cc:407] Connection header received
0806 17:03:00.024099 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:00.024105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:00.024177 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:00.024262 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:00.026226 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.026975 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.029066 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.029297 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.029412 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:00.033599 (+ 4187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.66:57341: BlockingRecv error: recv got EOF from 127.9.111.66:57341 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 17:03:00.034350 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:00.022082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.66:57341)
0806 17:03:00.022965 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:00.023026 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:00.023862 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:00.024502 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:00.024514 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:00.025102 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:00.025933 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:00.025967 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.027155 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.027162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:00.028856 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:00.028873 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.029510 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.029518 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:00.029794 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:00.030550 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:00.030579 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:00.033380 (+ 2801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":126,"threads_started":1}
W20260806 17:03:00.034989 18946 heartbeater.cc:644] Failed to heartbeat to 127.9.111.126:38999 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.111.126:38999: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Aug 06 17:03:00 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:00 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:00.040702 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:00.035476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.66:33749)
0806 17:03:00.035598 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:00.035615 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:00.035706 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:00.035973 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:00.035976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:00.036150 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:00.036248 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:00.036254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.037073 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.037077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:00.037630 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:00.037638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.038153 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.038156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:00.038246 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:00.038766 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:00.038778 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:00.040555 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 17:03:00.041092 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:00.035579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:33749 (local address 127.9.111.126:38999)
0806 17:03:00.035707 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:00.035712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:00.035727 (+ 15us) server_negotiation.cc:407] Connection header received
0806 17:03:00.035825 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:00.035827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:00.035867 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:00.035954 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:00.036374 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.036969 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.037751 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.038029 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.038154 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:00.040676 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.66:33749: BlockingRecv error: recv got EOF from 127.9.111.66:33749 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 17:03:00 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:00 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:00.047257 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:00.041245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.66:60201)
0806 17:03:00.041474 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:00.041495 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:00.041578 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:00.041834 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:00.041837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:00.042007 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:00.042111 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:00.042118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.042959 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.042963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:00.043904 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:00.043919 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.044448 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.044452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:00.044539 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:00.045104 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:00.045117 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:00.047092 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131}
W20260806 17:03:00.047362 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:00.041432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:60201 (local address 127.9.111.126:38999)
0806 17:03:00.041547 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:00.041550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:00.041563 (+ 13us) server_negotiation.cc:407] Connection header received
0806 17:03:00.041666 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:00.041668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:00.041711 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:00.041775 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:00.042248 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.042817 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.044057 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.044275 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.044443 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:00.047237 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.66:60201: BlockingRecv error: recv got EOF from 127.9.111.66:60201 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 17:03:00.047539 18946 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 17:03:00.148420 18949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:03:00.148766 18949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:03:00.148849 18949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:03:00.153098 18949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:03:00.153211 18949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:03:00.153303 18949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:03:00.158986 18949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/krb5kdc/oryx_127.9.111.67.keytab
--principal=oryx/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:36581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=42251
--tserver_master_addrs=127.9.111.126:38999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:03:00.160188 18949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:03:00.161337 18949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:03:00.171315 18955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:00.171315 18954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:00.171319 18958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:03:00.171765 18949 server_base.cc:1034] running on GCE node
I20260806 17:03:00.172206 18949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:03:00.172856 18949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:03:00.174026 18949 hybrid_clock.cc:648] HybridClock initialized: now 1786035780174010 us; error 36 us; skew 500 ppm
Aug 06 17:03:00 dist-test-slave-69rg krb5kdc[18057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035780, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:03:00.177280 18949 init.cc:377] Logged in from keytab as oryx/127.9.111.67@KRBTEST.COM (short username oryx)
I20260806 17:03:00.178378 18949 webserver.cc:460] Webserver started at http://127.9.111.67:42251/ using document root <none> and password file <none>
I20260806 17:03:00.178972 18949 fs_manager.cc:359] Metadata directory not provided
I20260806 17:03:00.179080 18949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:03:00.183164 18949 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 17:03:00.185395 18966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:00.186767 18949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:03:00.186903 18949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "a0d32cae4c5f41629336210f657f06d3"
format_stamp: "Formatted at 2026-08-06 17:02:59 on dist-test-slave-69rg"
server_key: "a9df30955923067cb9609cd9ca9556ce"
server_key_iv: "4310c873edc0c68fa3b6d58744b61803"
server_key_version: "encryptionkey@0"
I20260806 17:03:00.187477 18949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:03:00.211436 18949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:03:00.214555 18949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:03:00.214794 18949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:03:00.215433 18949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:03:00.216324 18949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:03:00.216372 18949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:00.216439 18949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:03:00.216471 18949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:00.226222 18949 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:36581
I20260806 17:03:00.226259 19081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:36581 every 8 connection(s)
I20260806 17:03:00.227295 18949 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:03:00.230425 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 18949
Aug 06 17:03:00 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:00 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
W20260806 17:03:00.239964 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:00.229363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:39687 (local address 127.9.111.126:38999)
0806 17:03:00.229536 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:00.229540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:00.230736 (+ 1196us) server_negotiation.cc:407] Connection header received
0806 17:03:00.231919 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:00.231922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:00.232049 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:00.232155 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:00.233554 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.234408 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.235855 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.236065 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.236199 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:00.239828 (+ 3629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.67:39687: BlockingRecv error: recv got EOF from 127.9.111.67:39687 (error 108)
Metrics: {"server-negotiator.queue_time_us":74,"spinlock_wait_cycles":17920}
I20260806 17:03:00.240741 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:00.229686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.67:39687)
0806 17:03:00.230412 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:00.230456 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:00.231654 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:00.232359 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:00.232367 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:00.232729 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:00.233385 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:00.233406 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.234527 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.234530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:00.235725 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:00.235735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.236222 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.236226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:00.236412 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:00.237006 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:00.237023 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:00.239704 (+ 2681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":133,"threads_started":1}
W20260806 17:03:00.241361 19082 heartbeater.cc:644] Failed to heartbeat to 127.9.111.126:38999 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.111.126:38999: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
I20260806 17:03:00.242272 18814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:00.231553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35226 (local address 127.9.111.126:38999)
0806 17:03:00.231707 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:00.231722 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:00.231738 (+ 16us) server_negotiation.cc:407] Connection header received
0806 17:03:00.231807 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:00.231813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:00.231875 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:00.231966 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:00.232989 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.233729 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.234936 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.235276 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.235387 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:00.237249 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:03:00.237256 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:03:00.237276 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:03:00.237308 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:03:00.239509 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:03:00.240058 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:03:00.240065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:03:00.240075 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:03:00.240274 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:03:00.240611 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:03:00.240615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:03:00.240617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:03:00.241174 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:03:00.241401 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:00.241743 (+ 342us) server_negotiation.cc:299] Negotiation successful
0806 17:03:00.242093 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63,"spinlock_wait_cycles":53888}
Aug 06 17:03:00 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:00 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:00.247928 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:00.242178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.67:34545)
0806 17:03:00.242332 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:00.242349 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:00.242493 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:00.242772 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:00.242775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:00.243026 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:00.243141 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:00.243147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.243975 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.243979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:00.244758 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:00.244768 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.245317 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.245321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:00.245424 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:00.245946 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:00.245958 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:00.247764 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 17:03:00.248019 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:00.242026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:34545 (local address 127.9.111.126:38999)
0806 17:03:00.242149 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:00.242154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:00.242413 (+ 259us) server_negotiation.cc:407] Connection header received
0806 17:03:00.242607 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:00.242610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:00.242665 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:00.242754 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:00.243265 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.243869 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.244887 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.245192 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.245317 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:00.247892 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.67:34545: BlockingRecv error: recv got EOF from 127.9.111.67:34545 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 06 17:03:00 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:00 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:00.253361 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:00.248628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.67:47817)
0806 17:03:00.248768 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:00.248787 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:00.248872 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:00.249067 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:00.249069 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:00.249210 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:00.249284 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:00.249289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.250032 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.250035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:00.250588 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:00.250595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.250995 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.250998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:00.251059 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:00.251519 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:00.251530 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:00.253244 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 17:03:00.253491 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:00.248731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:47817 (local address 127.9.111.126:38999)
0806 17:03:00.248886 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:00.248889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:00.248902 (+ 13us) server_negotiation.cc:407] Connection header received
0806 17:03:00.248935 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:00.248937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:00.248970 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:00.249035 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:00.249381 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.249940 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.250693 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.250894 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.250974 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:00.253376 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.67:47817: BlockingRecv error: recv got EOF from 127.9.111.67:47817 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260806 17:03:00.253679 19082 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 17:03:00 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:00 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:00.851866 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:00.839530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.65:56975)
0806 17:03:00.839865 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:00.839881 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:00.839988 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:00.840450 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:00.840452 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:00.840655 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:00.840759 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:00.840765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.841731 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.841734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:00.842686 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:00.842699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.843423 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.843430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:00.843547 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:00.849510 (+ 5963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:00.849526 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:00.851724 (+ 2198us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":85,"threads_started":1}
W20260806 17:03:00.852237 19088 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:00.839614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:56975 (local address 127.9.111.126:38999)
0806 17:03:00.839930 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:00.839934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:00.839950 (+ 16us) server_negotiation.cc:407] Connection header received
0806 17:03:00.840127 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:00.840130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:00.840182 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:00.840343 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:00.840920 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.841546 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.842837 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:00.843240 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:00.843352 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:00.852004 (+ 8652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.65:56975: BlockingRecv error: recv got EOF from 127.9.111.65:56975 (error 108)
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":91,"threads_started":1}
Aug 06 17:03:01 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:01 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:01.054232 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:01.048198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.66:50645)
0806 17:03:01.048494 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:01.048510 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:01.048693 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:01.048983 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:01.048986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:01.049208 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:01.049306 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:01.049312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:01.050177 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:01.050182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:01.050791 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:01.050801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:01.051309 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:01.051312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:01.051395 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:01.051950 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:01.051962 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:01.054067 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":87,"threads_started":1}
W20260806 17:03:01.054399 19088 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:01.048333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:50645 (local address 127.9.111.126:38999)
0806 17:03:01.048484 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:01.048489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:01.048602 (+ 113us) server_negotiation.cc:407] Connection header received
0806 17:03:01.048806 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:01.048810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:01.048859 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:01.048956 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:01.049437 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:01.050056 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:01.050924 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:01.051188 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:01.051303 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:01.054224 (+ 2921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.66:50645: BlockingRecv error: recv got EOF from 127.9.111.66:50645 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 17:03:01 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:01 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:01.260680 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:01.254532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.67:38413)
0806 17:03:01.254858 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:01.254874 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:01.254981 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:01.255304 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:01.255307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:01.255503 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:01.255637 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:01.255644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:01.256511 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:01.256514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:01.257099 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:01.257108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:01.257602 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:01.257605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:01.257676 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:01.258437 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:01.258452 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:01.260556 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":101,"threads_started":1}
W20260806 17:03:01.260921 19088 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:01.254752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:38413 (local address 127.9.111.126:38999)
0806 17:03:01.254919 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:01.254922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:01.254940 (+ 18us) server_negotiation.cc:407] Connection header received
0806 17:03:01.255082 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:01.255085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:01.255131 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:01.255206 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:01.255794 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:01.256399 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:01.257250 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:01.257487 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:01.257581 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:01.260791 (+ 3210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.67:38413: BlockingRecv error: recv got EOF from 127.9.111.67:38413 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 06 17:03:01 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:01 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:01.858739 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:01.852871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.65:41211)
0806 17:03:01.853152 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:01.853178 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:01.853284 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:01.853550 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:01.853552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:01.853736 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:01.853901 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:01.853909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:01.854823 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:01.854828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:01.855418 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:01.855427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:01.855921 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:01.855925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:01.856017 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:01.856558 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:01.856570 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:01.858519 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":103,"threads_started":1}
W20260806 17:03:01.858860 19093 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:01.853006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:41211 (local address 127.9.111.126:38999)
0806 17:03:01.853294 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:01.853298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:01.853343 (+ 45us) server_negotiation.cc:407] Connection header received
0806 17:03:01.853388 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:01.853390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:01.853435 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:01.853507 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:01.854034 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:01.854682 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:01.855553 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:01.855798 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:01.855913 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:01.858693 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.65:41211: BlockingRecv error: recv got EOF from 127.9.111.65:41211 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":121,"threads_started":1}
Aug 06 17:03:02 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:02 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:02.061786 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:02.055194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.66:58509)
0806 17:03:02.055546 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:02.055560 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:02.055655 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:02.055923 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:02.055925 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:02.056108 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:02.056202 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:02.056207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:02.057010 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:02.057012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:02.057583 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:02.057591 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:02.058347 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:02.058350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:02.058410 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:02.058923 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:02.058934 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:02.061528 (+ 2594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":129,"threads_started":1}
W20260806 17:03:02.061900 19093 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:02.055420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:58509 (local address 127.9.111.126:38999)
0806 17:03:02.055635 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:02.055639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:02.055660 (+ 21us) server_negotiation.cc:407] Connection header received
0806 17:03:02.055752 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:02.055755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:02.055799 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:02.055880 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:02.056339 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:02.056896 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:02.057701 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:02.058226 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:02.058322 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:02.061713 (+ 3391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.66:58509: BlockingRecv error: recv got EOF from 127.9.111.66:58509 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 06 17:03:02 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:02 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:02.267359 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:02.261512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.67:58253)
0806 17:03:02.261880 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:02.261896 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:02.262038 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:02.262352 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:02.262355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:02.262570 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:02.262660 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:02.262666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:02.263437 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:02.263443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:02.264004 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:02.264012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:02.264499 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:02.264503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:02.264591 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:02.265139 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:02.265150 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:02.267157 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":108,"threads_started":1}
W20260806 17:03:02.267453 19093 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:02.261703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:58253 (local address 127.9.111.126:38999)
0806 17:03:02.261887 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:02.261892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:02.261976 (+ 84us) server_negotiation.cc:407] Connection header received
0806 17:03:02.262145 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:02.262148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:02.262203 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:02.262327 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:02.262785 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:02.263321 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:02.264128 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:02.264374 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:02.264495 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:02.267325 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.67:58253: BlockingRecv error: recv got EOF from 127.9.111.67:58253 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 06 17:03:02 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:02 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:02.865975 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:02.859714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.65:51775)
0806 17:03:02.860072 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:02.860087 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:02.860199 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:02.860530 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:02.860534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:02.860736 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:02.860839 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:02.860845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:02.861674 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:02.861678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:02.862410 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:02.862421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:02.862885 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:02.862887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:02.862949 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:02.863554 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:02.863566 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:02.865767 (+ 2201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":140,"threads_started":1}
W20260806 17:03:02.866196 19096 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:02.859766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:51775 (local address 127.9.111.126:38999)
0806 17:03:02.860046 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:02.860050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:02.860188 (+ 138us) server_negotiation.cc:407] Connection header received
0806 17:03:02.860275 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:02.860278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:02.860328 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:02.860454 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:02.860971 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:02.861549 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:02.862540 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:02.862779 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:02.862863 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:02.866002 (+ 3139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.65:51775: BlockingRecv error: recv got EOF from 127.9.111.65:51775 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":96,"threads_started":1}
Aug 06 17:03:03 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:03 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:03.073094 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:03.062716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.66:43997)
0806 17:03:03.063002 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:03.063016 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:03.063109 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:03.063382 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:03.063384 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:03.063572 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:03.063740 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:03.063749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:03.064729 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:03.064733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:03.065355 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:03.065368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:03.065996 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:03.065998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:03.066070 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:03.070845 (+ 4775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:03.070858 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:03.072967 (+ 2109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":102,"threads_started":1}
W20260806 17:03:03.073367 19096 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:03.062900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:43997 (local address 127.9.111.126:38999)
0806 17:03:03.063056 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:03.063059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:03.063075 (+ 16us) server_negotiation.cc:407] Connection header received
0806 17:03:03.063208 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:03.063210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:03.063257 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:03.063340 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:03.063875 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:03.064596 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:03.065527 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:03.065775 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:03.065979 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:03.073235 (+ 7256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.66:43997: BlockingRecv error: recv got EOF from 127.9.111.66:43997 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 17:03:03 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:03 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:03.273996 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:03.268213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.67:49743)
0806 17:03:03.268487 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:03.268501 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:03.268601 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:03.268856 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:03.268858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:03.269047 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:03.269154 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:03.269160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:03.270119 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:03.270124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:03.270702 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:03.270711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:03.271190 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:03.271193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:03.271282 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:03.271804 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:03.271816 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:03.273809 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":108,"threads_started":1}
W20260806 17:03:03.274211 19096 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:03.268290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:49743 (local address 127.9.111.126:38999)
0806 17:03:03.268430 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:03.268433 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:03.268568 (+ 135us) server_negotiation.cc:407] Connection header received
0806 17:03:03.268678 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:03.268680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:03.268743 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:03.268814 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:03.269281 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:03.269991 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:03.270837 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:03.271072 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:03.271185 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:03.274067 (+ 2882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.67:49743: BlockingRecv error: recv got EOF from 127.9.111.67:49743 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 17:03:03 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:03 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:03.872819 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:03.866775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.65:34309)
0806 17:03:03.867054 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:03.867068 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:03.867163 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:03.867479 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:03.867482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:03.867696 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:03.867802 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:03.867808 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:03.868680 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:03.868683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:03.869482 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:03.869493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:03.870068 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:03.870071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:03.870150 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:03.870787 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:03.870801 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:03.872694 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":105,"threads_started":1}
W20260806 17:03:03.873088 19101 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:03.866874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:34309 (local address 127.9.111.126:38999)
0806 17:03:03.867127 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:03.867130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:03.867143 (+ 13us) server_negotiation.cc:407] Connection header received
0806 17:03:03.867249 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:03.867251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:03.867300 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:03.867382 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:03.867935 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:03.868551 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:03.869615 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:03.869885 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:03.870026 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:03.872937 (+ 2911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.65:34309: BlockingRecv error: recv got EOF from 127.9.111.65:34309 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":101,"threads_started":1}
Aug 06 17:03:04 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:04 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:04.079489 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:04.073909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.66:51965)
0806 17:03:04.074195 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:04.074208 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:04.074311 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:04.074541 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:04.074543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:04.074727 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:04.074823 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:04.074828 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:04.075554 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:04.075556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:04.076192 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:04.076201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:04.076646 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:04.076648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:04.076708 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:04.077309 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:04.077347 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:04.079374 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":117,"threads_started":1}
W20260806 17:03:04.079659 19101 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:04.074041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:51965 (local address 127.9.111.126:38999)
0806 17:03:04.074189 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:04.074193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:04.074278 (+ 85us) server_negotiation.cc:407] Connection header received
0806 17:03:04.074388 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:04.074390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:04.074433 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:04.074503 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:04.074929 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:04.075455 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:04.076318 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:04.076542 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:04.076634 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:04.079522 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.66:51965: BlockingRecv error: recv got EOF from 127.9.111.66:51965 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 06 17:03:04 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:04 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:04.283243 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:04.274925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.67:47819)
0806 17:03:04.275479 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:04.275494 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:04.275701 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:04.276353 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:04.276356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:04.276748 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:04.276863 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:04.276870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:04.278082 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:04.278085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:04.278721 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:04.278731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:04.279338 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:04.279341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:04.279418 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:04.280400 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:04.280414 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:04.283113 (+ 2699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":150,"threads_started":1}
W20260806 17:03:04.283450 19101 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:04.275090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:47819 (local address 127.9.111.126:38999)
0806 17:03:04.275292 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:04.275299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:04.275655 (+ 356us) server_negotiation.cc:407] Connection header received
0806 17:03:04.275885 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:04.275888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:04.275946 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:04.276045 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:04.277217 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:04.277937 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:04.278886 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:04.279189 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:04.279288 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:04.283296 (+ 4008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.67:47819: BlockingRecv error: recv got EOF from 127.9.111.67:47819 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Aug 06 17:03:04 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:04 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:04.880271 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:04.873662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.65:60293)
0806 17:03:04.874047 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:04.874067 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:04.874172 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:04.874585 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:04.874591 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:04.874850 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:04.874951 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:04.874957 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:04.875789 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:04.875792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:04.876481 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:04.876492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:04.877037 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:04.877041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:04.877111 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:04.877914 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:04.877926 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:04.880084 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":116,"threads_started":1}
W20260806 17:03:04.880421 19105 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:04.873750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:60293 (local address 127.9.111.126:38999)
0806 17:03:04.874056 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:04.874061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:04.874151 (+ 90us) server_negotiation.cc:407] Connection header received
0806 17:03:04.874303 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:04.874305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:04.874374 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:04.874471 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:04.875074 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:04.875654 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:04.876631 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:04.876895 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:04.876990 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:04.880285 (+ 3295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.65:60293: BlockingRecv error: recv got EOF from 127.9.111.65:60293 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":111,"threads_started":1}
Aug 06 17:03:05 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:05 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:05.088161 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:05.080551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.66:32969)
0806 17:03:05.080952 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:05.080971 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:05.081131 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:05.081496 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:05.081500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:05.081737 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:05.081992 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:05.082005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:05.083299 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:05.083304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:05.084124 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:05.084138 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:05.084784 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:05.084788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:05.084881 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:05.085606 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:05.085626 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:05.088035 (+ 2409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":111,"threads_started":1}
W20260806 17:03:05.088394 19105 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:05.080792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:32969 (local address 127.9.111.126:38999)
0806 17:03:05.080967 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:05.080973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:05.081065 (+ 92us) server_negotiation.cc:407] Connection header received
0806 17:03:05.081308 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:05.081311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:05.081360 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:05.081461 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:05.082148 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:05.083112 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:05.084299 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:05.084619 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:05.084780 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:05.088220 (+ 3440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.66:32969: BlockingRecv error: recv got EOF from 127.9.111.66:32969 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 06 17:03:05 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:05 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:05.290418 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:05.284383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.67:60965)
0806 17:03:05.284702 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:05.284726 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:05.284853 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:05.285126 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:05.285130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:05.285389 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:05.285484 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:05.285491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:05.286458 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:05.286463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:05.287067 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:05.287076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:05.287566 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:05.287570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:05.287658 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:05.288264 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:05.288277 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:05.290234 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":123,"threads_started":1}
W20260806 17:03:05.290508 19105 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:05.284704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:60965 (local address 127.9.111.126:38999)
0806 17:03:05.284855 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:05.284861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:05.284877 (+ 16us) server_negotiation.cc:407] Connection header received
0806 17:03:05.284948 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:05.284951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:05.285031 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:05.285104 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:05.285614 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:05.286345 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:05.287194 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:05.287444 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:05.287560 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:05.290384 (+ 2824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.67:60965: BlockingRecv error: recv got EOF from 127.9.111.67:60965 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 17:03:05 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:05 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:05.895718 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:05.881303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.65:52477)
0806 17:03:05.881756 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:05.881771 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:05.881933 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:05.882466 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:05.882469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:05.882674 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:05.882778 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:05.882784 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:05.883720 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:05.883724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:05.884315 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:05.884325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:05.884822 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:05.884827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:05.884921 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:05.893091 (+ 8170us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:05.893108 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:05.895589 (+ 2481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":115,"threads_started":1}
W20260806 17:03:05.895967 19109 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:05.881411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:52477 (local address 127.9.111.126:38999)
0806 17:03:05.881697 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:05.881704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:05.881843 (+ 139us) server_negotiation.cc:407] Connection header received
0806 17:03:05.882135 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:05.882140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:05.882235 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:05.882424 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:05.882914 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:05.883559 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:05.884430 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:05.884659 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:05.884747 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:05.895822 (+ 11075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.65:52477: BlockingRecv error: recv got EOF from 127.9.111.65:52477 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
Aug 06 17:03:06 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:06 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:06.095235 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:06.089082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.66:57493)
0806 17:03:06.089363 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:06.089380 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:06.089485 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:06.089678 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:06.089679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:06.089939 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:06.090036 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:06.090043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:06.090842 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:06.090845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:06.091478 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:06.091487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:06.092062 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:06.092064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:06.092130 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:06.092710 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:06.092722 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:06.095097 (+ 2375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":168,"threads_started":1}
W20260806 17:03:06.095475 19109 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:06.089153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:57493 (local address 127.9.111.126:38999)
0806 17:03:06.089327 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:06.089331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:06.089434 (+ 103us) server_negotiation.cc:407] Connection header received
0806 17:03:06.089551 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:06.089553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:06.089596 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:06.089664 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:06.090148 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:06.090725 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:06.091660 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:06.091940 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:06.092035 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:06.095341 (+ 3306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.66:57493: BlockingRecv error: recv got EOF from 127.9.111.66:57493 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 06 17:03:06 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:06 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:06.297298 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:06.291388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.67:41599)
0806 17:03:06.291695 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:06.291713 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:06.291824 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:06.292102 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:06.292105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:06.292344 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:06.292441 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:06.292447 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:06.293270 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:06.293273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:06.293879 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:06.293890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:06.294371 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:06.294374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:06.294440 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:06.295140 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:06.295154 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:06.297160 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":138,"threads_started":1}
W20260806 17:03:06.297632 19109 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:06.291540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:41599 (local address 127.9.111.126:38999)
0806 17:03:06.291684 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:06.291688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:06.291845 (+ 157us) server_negotiation.cc:407] Connection header received
0806 17:03:06.291898 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:06.291900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:06.291952 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:06.292034 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:06.292574 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:06.293120 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:06.293987 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:06.294243 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:06.294340 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:06.297487 (+ 3147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.67:41599: BlockingRecv error: recv got EOF from 127.9.111.67:41599 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 17:03:06 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:06 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:06.903177 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:06.896800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.65:39163)
0806 17:03:06.897240 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:06.897255 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:06.897404 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:06.897773 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:06.897776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:06.898005 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:06.898122 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:06.898140 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:06.898977 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:06.898980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:06.899606 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:06.899616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:06.900102 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:06.900105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:06.900211 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:06.900796 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:06.900809 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:06.903021 (+ 2212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":152,"threads_started":1}
W20260806 17:03:06.903532 19113 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:06.897108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:39163 (local address 127.9.111.126:38999)
0806 17:03:06.897378 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:06.897381 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:06.897398 (+ 17us) server_negotiation.cc:407] Connection header received
0806 17:03:06.897538 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:06.897540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:06.897587 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:06.897699 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:06.898306 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:06.898857 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:06.899746 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:06.899980 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:06.900068 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:06.903375 (+ 3307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.65:39163: BlockingRecv error: recv got EOF from 127.9.111.65:39163 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":99,"threads_started":1}
Aug 06 17:03:07 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:07 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:07.103768 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:07.096321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.66:48311)
0806 17:03:07.096780 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:07.096799 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:07.096977 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:07.097276 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:07.097279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:07.097505 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:07.097602 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:07.097608 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:07.098512 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:07.098517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:07.099093 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:07.099102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:07.099681 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:07.099685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:07.099770 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:07.100355 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:07.100368 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:07.103534 (+ 3166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":97,"threads_started":1}
W20260806 17:03:07.103838 19113 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:07.096508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:48311 (local address 127.9.111.126:38999)
0806 17:03:07.096756 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:07.096761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:07.096897 (+ 136us) server_negotiation.cc:407] Connection header received
0806 17:03:07.097090 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:07.097094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:07.097150 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:07.097248 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:07.097744 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:07.098378 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:07.099233 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:07.099487 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:07.099678 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:07.103714 (+ 4036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.66:48311: BlockingRecv error: recv got EOF from 127.9.111.66:48311 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 06 17:03:07 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:07 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:07.304875 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:07.298478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.67:42137)
0806 17:03:07.298857 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:07.298872 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:07.298986 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:07.299294 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:07.299297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:07.299511 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:07.299606 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:07.299640 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:07.300435 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:07.300438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:07.301036 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:07.301045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:07.301579 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:07.301583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:07.301687 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:07.302251 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:07.302268 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:07.304689 (+ 2421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":110,"threads_started":1}
W20260806 17:03:07.304947 19113 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:07.298584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:42137 (local address 127.9.111.126:38999)
0806 17:03:07.298758 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:07.298762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:07.298958 (+ 196us) server_negotiation.cc:407] Connection header received
0806 17:03:07.299083 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:07.299086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:07.299136 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:07.299221 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:07.299763 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:07.300313 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:07.301215 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:07.301455 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:07.301546 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:07.304833 (+ 3287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.67:42137: BlockingRecv error: recv got EOF from 127.9.111.67:42137 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 06 17:03:07 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:07 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:07.910744 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:07.904360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.65:46121)
0806 17:03:07.904681 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:07.904702 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:07.904830 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:07.905244 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:07.905248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:07.905474 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:07.905622 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:07.905632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:07.906595 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:07.906598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:07.907204 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:07.907214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:07.907740 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:07.907743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:07.907841 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:07.908422 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:07.908438 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:07.910614 (+ 2176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":118,"threads_started":1}
W20260806 17:03:07.910983 19117 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:07.904494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:46121 (local address 127.9.111.126:38999)
0806 17:03:07.904765 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:07.904771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:07.904788 (+ 17us) server_negotiation.cc:407] Connection header received
0806 17:03:07.904962 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:07.904965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:07.905027 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:07.905133 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:07.905797 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:07.906455 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:07.907354 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:07.907589 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:07.907716 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:07.910806 (+ 3090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.65:46121: BlockingRecv error: recv got EOF from 127.9.111.65:46121 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
Aug 06 17:03:08 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:08 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:08.110559 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:08.104673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.66:48879)
0806 17:03:08.104991 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:08.105006 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:08.105101 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:08.105371 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:08.105373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:08.105563 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:08.105663 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:08.105668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:08.106615 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:08.106619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:08.107182 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:08.107191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:08.107724 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:08.107728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:08.107800 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:08.108427 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:08.108439 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:08.110434 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":91,"threads_started":1}
W20260806 17:03:08.110827 19117 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:08.104852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:48879 (local address 127.9.111.126:38999)
0806 17:03:08.105031 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:08.105035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:08.105053 (+ 18us) server_negotiation.cc:407] Connection header received
0806 17:03:08.105209 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:08.105212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:08.105258 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:08.105327 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:08.105779 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:08.106492 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:08.107332 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:08.107570 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:08.107661 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:08.110698 (+ 3037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.66:48879: BlockingRecv error: recv got EOF from 127.9.111.66:48879 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 06 17:03:08 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:08 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:08.314162 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:08.306515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.67:52681)
0806 17:03:08.307270 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:08.307290 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:08.307493 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:08.307841 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:08.307845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:08.308087 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:08.308202 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:08.308209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:08.309066 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:08.309071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:08.309718 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:08.309728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:08.310299 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:08.310303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:08.310402 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:08.311047 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:08.311059 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:08.314035 (+ 2976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":105,"threads_started":1}
W20260806 17:03:08.314417 19117 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:08.306693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:52681 (local address 127.9.111.126:38999)
0806 17:03:08.307263 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:08.307268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:08.307407 (+ 139us) server_negotiation.cc:407] Connection header received
0806 17:03:08.307617 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:08.307631 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:08.307698 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:08.307807 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:08.308343 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:08.308935 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:08.309882 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:08.310152 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:08.310289 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:08.314296 (+ 4007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.67:52681: BlockingRecv error: recv got EOF from 127.9.111.67:52681 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Aug 06 17:03:08 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:08 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:08.919278 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:08.911887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.65:32931)
0806 17:03:08.912504 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:08.912520 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:08.912688 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:08.913311 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:08.913314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:08.913613 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:08.913732 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:08.913739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:08.914832 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:08.914835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:08.915455 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:08.915465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:08.916010 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:08.916014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:08.916090 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:08.916700 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:08.916713 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:08.919110 (+ 2397us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":139,"threads_started":1}
W20260806 17:03:08.919672 19123 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:08.912295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:32931 (local address 127.9.111.126:38999)
0806 17:03:08.912718 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:08.912722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:08.912743 (+ 21us) server_negotiation.cc:407] Connection header received
0806 17:03:08.912837 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:08.912840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:08.912914 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:08.913226 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:08.913901 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:08.914711 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:08.915595 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:08.915843 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:08.915933 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:08.919521 (+ 3588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.65:32931: BlockingRecv error: recv got EOF from 127.9.111.65:32931 (error 108)
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":80,"threads_started":1}
Aug 06 17:03:09 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:09 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:09.117532 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:09.111446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.66:52467)
0806 17:03:09.111783 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:09.111797 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:09.111911 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:09.112183 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:09.112186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:09.112382 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:09.112493 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:09.112499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:09.113352 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:09.113355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:09.113978 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:09.113988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:09.114603 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:09.114606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:09.114690 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:09.115286 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:09.115298 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:09.117385 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":133,"threads_started":1}
W20260806 17:03:09.117754 19123 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:09.111501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:52467 (local address 127.9.111.126:38999)
0806 17:03:09.111663 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:09.111668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:09.111903 (+ 235us) server_negotiation.cc:407] Connection header received
0806 17:03:09.111996 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:09.111999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:09.112053 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:09.112120 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:09.112649 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:09.113221 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:09.114182 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:09.114472 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:09.114562 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:09.117610 (+ 3048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.66:52467: BlockingRecv error: recv got EOF from 127.9.111.66:52467 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 17:03:09 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:09 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:09.322495 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:09.315396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.67:32997)
0806 17:03:09.315769 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:09.315786 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:09.315963 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:09.316488 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:09.316491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:09.316732 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:09.316857 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:09.316865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:09.317724 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:09.317728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:09.318423 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:09.318436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:09.318998 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:09.319002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:09.319071 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:09.319746 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:09.319760 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:09.322333 (+ 2573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":135,"threads_started":1}
W20260806 17:03:09.322695 19123 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:09.315555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:32997 (local address 127.9.111.126:38999)
0806 17:03:09.315722 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:09.315726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:09.315887 (+ 161us) server_negotiation.cc:407] Connection header received
0806 17:03:09.316157 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:09.316160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:09.316222 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:09.316329 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:09.317040 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:09.317587 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:09.318579 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:09.318828 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:09.318927 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:09.322531 (+ 3604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.67:32997: BlockingRecv error: recv got EOF from 127.9.111.67:32997 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 06 17:03:09 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:09 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:09.927456 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:09.920451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.65:53281)
0806 17:03:09.920878 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:09.920892 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:09.920984 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:09.921335 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:09.921339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:09.921607 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:09.921766 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:09.921774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:09.922791 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:09.922795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:09.923406 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:09.923417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:09.924223 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:09.924226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:09.924307 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:09.924912 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:09.924925 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:09.927324 (+ 2399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":128,"threads_started":1}
W20260806 17:03:09.927742 19127 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:09.920493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:53281 (local address 127.9.111.126:38999)
0806 17:03:09.920931 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:09.920936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:09.920953 (+ 17us) server_negotiation.cc:407] Connection header received
0806 17:03:09.921093 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:09.921097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:09.921162 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:09.921240 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:09.922040 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:09.922620 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:09.923611 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:09.923880 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:09.924144 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:09.927586 (+ 3442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.65:53281: BlockingRecv error: recv got EOF from 127.9.111.65:53281 (error 108)
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":99,"threads_started":1}
Aug 06 17:03:10 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:10 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:10.125344 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:10.118597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.66:34903)
0806 17:03:10.118953 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:10.118971 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:10.119087 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:10.119400 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:10.119403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:10.119923 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:10.120041 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:10.120048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:10.121111 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:10.121116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:10.121723 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:10.121733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:10.122253 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:10.122256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:10.122338 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:10.122910 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:10.122921 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:10.125145 (+ 2224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":104,"threads_started":1}
W20260806 17:03:10.125435 19127 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:10.118878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:34903 (local address 127.9.111.126:38999)
0806 17:03:10.119067 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:10.119071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:10.119089 (+ 18us) server_negotiation.cc:407] Connection header received
0806 17:03:10.119168 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:10.119172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:10.119232 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:10.119311 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:10.120228 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:10.120982 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:10.121882 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:10.122123 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:10.122247 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:10.125309 (+ 3062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.66:34903: BlockingRecv error: recv got EOF from 127.9.111.66:34903 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 17:03:10 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:10 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:10.330103 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:10.323563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.67:57781)
0806 17:03:10.323902 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:10.323916 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:10.324008 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:10.324413 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:10.324415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:10.324639 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:10.324761 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:10.324767 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:10.325699 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:10.325702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:10.326445 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:10.326457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:10.326995 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:10.326999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:10.327065 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:10.327936 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:10.327949 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:10.329975 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":81,"threads_started":1}
W20260806 17:03:10.330386 19127 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:10.323791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:57781 (local address 127.9.111.126:38999)
0806 17:03:10.323958 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:10.323962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:10.323977 (+ 15us) server_negotiation.cc:407] Connection header received
0806 17:03:10.324139 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:10.324142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:10.324230 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:10.324313 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:10.324919 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:10.325530 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:10.326603 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:10.326866 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:10.326954 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:10.330232 (+ 3278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.67:57781: BlockingRecv error: recv got EOF from 127.9.111.67:57781 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 06 17:03:10 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:10 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:10.939496 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:10.928582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.65:42005)
0806 17:03:10.928987 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:10.929004 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:10.929140 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:10.929422 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:10.929424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:10.929609 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:10.929705 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:10.929710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:10.930537 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:10.930540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:10.931106 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:10.931115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:10.931639 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:10.931642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:10.931729 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:10.937085 (+ 5356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:10.937101 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:10.939331 (+ 2230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":116,"threads_started":1}
W20260806 17:03:10.939810 19131 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:10.928784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:42005 (local address 127.9.111.126:38999)
0806 17:03:10.929054 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:10.929058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:10.929074 (+ 16us) server_negotiation.cc:407] Connection header received
0806 17:03:10.929256 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:10.929258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:10.929305 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:10.929386 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:10.929807 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:10.930410 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:10.931271 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:10.931506 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:10.931590 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:10.939609 (+ 8019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.65:42005: BlockingRecv error: recv got EOF from 127.9.111.65:42005 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":76,"threads_started":1}
Aug 06 17:03:11 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:11 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:11.134717 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:11.126614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.66:47431)
0806 17:03:11.127057 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:11.127070 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:11.127177 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:11.127486 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:11.127489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:11.127727 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:11.127823 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:11.127829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:11.128647 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:11.128652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:11.129636 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:11.129648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:11.130319 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:11.130323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:11.130399 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:11.131038 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:11.131061 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:11.134512 (+ 3451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":146,"threads_started":1}
W20260806 17:03:11.134857 19131 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:11.126766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:47431 (local address 127.9.111.126:38999)
0806 17:03:11.126947 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:11.126952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:11.127161 (+ 209us) server_negotiation.cc:407] Connection header received
0806 17:03:11.127307 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:11.127309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:11.127359 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:11.127457 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:11.127954 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:11.128527 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:11.129913 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:11.130172 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:11.130319 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:11.134724 (+ 4405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.66:47431: BlockingRecv error: recv got EOF from 127.9.111.66:47431 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 17:03:11 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:11 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:11.337652 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:11.331011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.67:48831)
0806 17:03:11.331532 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:11.331546 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:11.331658 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:11.331938 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:11.331942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:11.332144 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:11.332249 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:11.332254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:11.333087 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:11.333090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:11.334220 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:11.334236 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:11.334834 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:11.334837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:11.334906 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:11.335500 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:11.335513 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:11.337512 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":129,"threads_started":1}
W20260806 17:03:11.337872 19131 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:11.331162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:48831 (local address 127.9.111.126:38999)
0806 17:03:11.331344 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:11.331349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:11.331647 (+ 298us) server_negotiation.cc:407] Connection header received
0806 17:03:11.331745 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:11.331748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:11.331799 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:11.331874 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:11.332419 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:11.332957 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:11.334391 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:11.334677 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:11.334769 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:11.337682 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.67:48831: BlockingRecv error: recv got EOF from 127.9.111.67:48831 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 17:03:11 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:11 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:11.947391 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:11.940495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.65:39993)
0806 17:03:11.940905 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:11.940926 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:11.941045 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:11.941296 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:11.941298 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:11.941502 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:11.941601 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:11.941606 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:11.942934 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:11.942937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:11.943537 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:11.943547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:11.944379 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:11.944382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:11.944472 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:11.945221 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:11.945235 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:11.947262 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":96,"threads_started":1}
W20260806 17:03:11.947850 19136 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:11.940785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:39993 (local address 127.9.111.126:38999)
0806 17:03:11.941066 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:11.941070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:11.941084 (+ 14us) server_negotiation.cc:407] Connection header received
0806 17:03:11.941127 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:11.941129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:11.941173 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:11.941262 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:11.941784 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:11.942753 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:11.943879 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:11.944201 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:11.944340 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:11.947681 (+ 3341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.65:39993: BlockingRecv error: recv got EOF from 127.9.111.65:39993 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":111,"threads_started":1}
Aug 06 17:03:12 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:12 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:12.143713 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:12.136104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.66:43441)
0806 17:03:12.136521 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:12.136541 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:12.136676 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:12.137049 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:12.137127 (+ 78us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:12.137464 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:12.137564 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:12.137571 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:12.138565 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:12.138569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:12.139550 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:12.139561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:12.140226 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:12.140230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:12.140318 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:12.140916 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:12.140929 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:12.143506 (+ 2577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":149,"threads_started":1}
W20260806 17:03:12.143952 19136 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:12.136248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:43441 (local address 127.9.111.126:38999)
0806 17:03:12.136455 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:12.136462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:12.136664 (+ 202us) server_negotiation.cc:407] Connection header received
0806 17:03:12.136750 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:12.136755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:12.136819 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:12.136989 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:12.137715 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:12.138454 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:12.139712 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:12.140112 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:12.140207 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:12.143696 (+ 3489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.66:43441: BlockingRecv error: recv got EOF from 127.9.111.66:43441 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 06 17:03:12 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:12 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:12.346980 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:12.338671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.67:35757)
0806 17:03:12.339069 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:12.339089 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:12.339250 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:12.339597 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:12.339600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:12.339918 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:12.340025 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:12.340032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:12.341103 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:12.341108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:12.341730 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:12.341740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:12.342614 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:12.342618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:12.342715 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:12.343501 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:12.343514 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:12.346793 (+ 3279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":145,"threads_started":1}
W20260806 17:03:12.347137 19136 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:12.338889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:35757 (local address 127.9.111.126:38999)
0806 17:03:12.339046 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:12.339052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:12.339206 (+ 154us) server_negotiation.cc:407] Connection header received
0806 17:03:12.339377 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:12.339381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:12.339444 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:12.339529 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:12.340199 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:12.340988 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:12.341946 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:12.342451 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:12.342606 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:12.347005 (+ 4399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.67:35757: BlockingRecv error: recv got EOF from 127.9.111.67:35757 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 06 17:03:12 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:12 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:12.954214 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:12.948382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.65:43185)
0806 17:03:12.948695 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:12.948716 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:12.948824 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:12.949120 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:12.949123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:12.949344 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:12.949451 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:12.949457 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:12.950281 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:12.950284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:12.950850 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:12.950859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:12.951260 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:12.951262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:12.951324 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:12.951827 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:12.951840 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:12.954027 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":96,"threads_started":1}
W20260806 17:03:12.954471 19140 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:12.948462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:43185 (local address 127.9.111.126:38999)
0806 17:03:12.948720 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:12.948723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:12.948778 (+ 55us) server_negotiation.cc:407] Connection header received
0806 17:03:12.948917 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:12.948919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:12.948963 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:12.949041 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:12.949567 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:12.950176 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:12.950954 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:12.951164 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:12.951237 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:12.954270 (+ 3033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.65:43185: BlockingRecv error: recv got EOF from 127.9.111.65:43185 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":66,"threads_started":1}
Aug 06 17:03:13 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:13 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:13.152213 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:13.144778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.66:52787)
0806 17:03:13.145071 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:13.145088 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:13.145200 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:13.145707 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:13.145710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:13.146015 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:13.146164 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:13.146171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:13.147381 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:13.147385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:13.148137 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:13.148199 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:13.148875 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:13.148878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:13.148970 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:13.149677 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:13.149694 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:13.152072 (+ 2378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":94,"threads_started":1}
W20260806 17:03:13.152487 19140 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:13.145289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:52787 (local address 127.9.111.126:38999)
0806 17:03:13.145429 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:13.145434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:13.145448 (+ 14us) server_negotiation.cc:407] Connection header received
0806 17:03:13.145489 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:13.145493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:13.145546 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:13.145631 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:13.146322 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:13.147215 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:13.148331 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:13.148732 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:13.148865 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:13.152314 (+ 3449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.66:52787: BlockingRecv error: recv got EOF from 127.9.111.66:52787 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 06 17:03:13 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:13 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:13.358979 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:13.349045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.67:47209)
0806 17:03:13.349667 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:13.349687 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:13.349938 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:13.350376 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:13.350379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:13.351306 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:13.351442 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:13.351450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:13.352828 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:13.352831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:13.353604 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:13.353621 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:13.354574 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:13.354585 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:13.354693 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:13.355630 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:13.355646 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:13.358803 (+ 3157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":94,"threads_started":1}
W20260806 17:03:13.359110 19140 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:13.349168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:47209 (local address 127.9.111.126:38999)
0806 17:03:13.349767 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:13.349774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:13.349799 (+ 25us) server_negotiation.cc:407] Connection header received
0806 17:03:13.350076 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:13.350080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:13.350139 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:13.350224 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:13.351888 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:13.352492 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:13.353885 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:13.354230 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:13.354326 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:13.358978 (+ 4652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.67:47209: BlockingRecv error: recv got EOF from 127.9.111.67:47209 (error 108)
Metrics: {"server-negotiator.queue_time_us":494}
Aug 06 17:03:13 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:13 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:13.961097 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:13.955043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.65:59595)
0806 17:03:13.955422 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:13.955446 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:13.955542 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:13.955871 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:13.955874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:13.956074 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:13.956169 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:13.956175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:13.956929 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:13.956932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:13.957512 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:13.957519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:13.958216 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:13.958219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:13.958293 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:13.958860 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:13.958872 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:13.960920 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":142,"threads_started":1}
W20260806 17:03:13.961259 19144 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:13.955187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:59595 (local address 127.9.111.126:38999)
0806 17:03:13.955506 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:13.955510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:13.955524 (+ 14us) server_negotiation.cc:407] Connection header received
0806 17:03:13.955683 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:13.955686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:13.955735 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:13.955822 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:13.956283 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:13.956834 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:13.957791 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:13.958087 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:13.958182 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:13.961097 (+ 2915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.65:59595: BlockingRecv error: recv got EOF from 127.9.111.65:59595 (error 108)
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":65,"threads_started":1}
Aug 06 17:03:14 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:14 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:14.159825 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:14.153655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.66:43177)
0806 17:03:14.154032 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:14.154049 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:14.154245 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:14.154544 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:14.154547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:14.154777 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:14.154871 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:14.154877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:14.155665 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:14.155670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:14.156258 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:14.156267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:14.156749 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:14.156753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:14.156842 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:14.157431 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:14.157443 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:14.159642 (+ 2199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":144,"threads_started":1}
W20260806 17:03:14.159940 19144 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:14.153866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:43177 (local address 127.9.111.126:38999)
0806 17:03:14.154038 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:14.154045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:14.154149 (+ 104us) server_negotiation.cc:407] Connection header received
0806 17:03:14.154363 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:14.154366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:14.154423 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:14.154515 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:14.155002 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:14.155543 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:14.156391 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:14.156621 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:14.156741 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:14.159824 (+ 3083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.66:43177: BlockingRecv error: recv got EOF from 127.9.111.66:43177 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 06 17:03:14 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:14 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.67@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:14.365950 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:14.359890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.67:58031)
0806 17:03:14.360279 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:14.360292 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:14.360433 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:14.360759 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:14.360762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:14.360994 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:14.361084 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:14.361089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:14.361891 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:14.361897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:14.362461 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:14.362469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:14.362938 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:14.362941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:14.363029 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:14.363612 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:14.363625 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:14.365761 (+ 2136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":88,"threads_started":1}
W20260806 17:03:14.366178 19144 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:14.360020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:58031 (local address 127.9.111.126:38999)
0806 17:03:14.360210 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:14.360216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:14.360383 (+ 167us) server_negotiation.cc:407] Connection header received
0806 17:03:14.360548 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:14.360553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:14.360637 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:14.360733 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:14.361214 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:14.361750 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:14.362589 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:14.362821 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:14.362934 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:14.366047 (+ 3113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.67:58031: BlockingRecv error: recv got EOF from 127.9.111.67:58031 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 06 17:03:14 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:14 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.65@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:14.967967 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:14.962213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.65:53757)
0806 17:03:14.962529 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:14.962544 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:14.962623 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:14.962894 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:14.962896 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:14.963092 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:14.963221 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:14.963228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:14.963973 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:14.963976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:14.964559 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:14.964568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:14.965057 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:14.965060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:14.965121 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:14.965636 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:14.965648 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:14.967835 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":121,"threads_started":1}
W20260806 17:03:14.968050 19148 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:14.962311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:53757 (local address 127.9.111.126:38999)
0806 17:03:14.962566 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:14.962569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:14.962607 (+ 38us) server_negotiation.cc:407] Connection header received
0806 17:03:14.962722 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:14.962724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:14.962770 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:14.962848 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:14.963344 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:14.963880 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:14.964685 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:14.964950 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:14.965039 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:14.967946 (+ 2907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.65:53757: BlockingRecv error: recv got EOF from 127.9.111.65:53757 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":98,"threads_started":1}
Aug 06 17:03:15 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 17:03:15 dist-test-slave-69rg krb5kdc[18057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.111.66@KRBTEST.COM for oryx/127.9.111.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 17:03:15.168498 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:15.161262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38999 (local address 127.9.111.66:60925)
0806 17:03:15.161580 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:15.161600 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:15.161713 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:15.162115 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:15.162119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:15.162448 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:15.162606 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:15.162615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:15.163534 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:15.163537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:15.164275 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:15.164288 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:15.164787 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:15.164791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:15.164858 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:15.165422 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:15.165434 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:15.168289 (+ 2855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.111.126:38999: Server oryx/127.9.111.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":92,"threads_started":1}
W20260806 17:03:15.169042 19148 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:15.161515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:60925 (local address 127.9.111.126:38999)
0806 17:03:15.161680 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:15.161686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:15.161701 (+ 15us) server_negotiation.cc:407] Connection header received
0806 17:03:15.161863 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:15.161866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:15.161924 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:15.162016 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:15.162773 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:15.163402 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:15.164434 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:15.164654 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:15.164742 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:15.168935 (+ 4193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.111.66:60925: BlockingRecv error: recv got EOF from 127.9.111.66:60925 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 17:03:15.233456 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 18658
I20260806 17:03:15.240546 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 18813
I20260806 17:03:15.248087 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 18949
I20260806 17:03:15.255028 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 18586
2026-08-06T17:03:15Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18145 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T17:03:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:03:15Z Disabled control of system clock
I20260806 17:03:15.291915 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:44459
--webserver_interface=127.9.111.126
--webserver_port=0
--builtin_ntp_servers=127.9.111.84:33227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:44459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 17:03:15.417186 19153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:03:15.417505 19153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:03:15.417543 19153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:03:15.421936 19153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:03:15.422016 19153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:03:15.422032 19153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:03:15.422051 19153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:03:15.422068 19153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:03:15.428449 19153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:33227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:44459
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:44459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:03:15.430171 19153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:03:15.431749 19153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 17:03:15.444613 19153 server_base.cc:1034] running on GCE node
W20260806 17:03:15.444506 19159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:15.444522 19158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:15.444525 19161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:03:15.446722 19153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:03:15.448467 19153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:03:15.449776 19153 hybrid_clock.cc:648] HybridClock initialized: now 1786035795449740 us; error 87 us; skew 500 ppm
I20260806 17:03:15.452108 19153 webserver.cc:460] Webserver started at http://127.9.111.126:37649/ using document root <none> and password file <none>
I20260806 17:03:15.452775 19153 fs_manager.cc:359] Metadata directory not provided
I20260806 17:03:15.452879 19153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:03:15.453141 19153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:03:15.455843 19153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "58cde0796ef74d9488ba4a5a82f2d4c4"
format_stamp: "Formatted at 2026-08-06 17:03:15 on dist-test-slave-69rg"
server_key: "5b019ddc88f9cc2d72c7eda0ffff4d84"
server_key_iv: "221bae6e2cf6b9c9379bc0f678903ae0"
server_key_version: "encryptionkey@0"
I20260806 17:03:15.456532 19153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "58cde0796ef74d9488ba4a5a82f2d4c4"
format_stamp: "Formatted at 2026-08-06 17:03:15 on dist-test-slave-69rg"
server_key: "5b019ddc88f9cc2d72c7eda0ffff4d84"
server_key_iv: "221bae6e2cf6b9c9379bc0f678903ae0"
server_key_version: "encryptionkey@0"
I20260806 17:03:15.460894 19153 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 17:03:15.464119 19167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:15.465715 19153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 17:03:15.465916 19153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "58cde0796ef74d9488ba4a5a82f2d4c4"
format_stamp: "Formatted at 2026-08-06 17:03:15 on dist-test-slave-69rg"
server_key: "5b019ddc88f9cc2d72c7eda0ffff4d84"
server_key_iv: "221bae6e2cf6b9c9379bc0f678903ae0"
server_key_version: "encryptionkey@0"
I20260806 17:03:15.466063 19153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:03:15.505579 19153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:03:15.506513 19153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:03:15.506758 19153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:03:15.515605 19153 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:44459
I20260806 17:03:15.515844 19220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:44459 every 8 connection(s)
I20260806 17:03:15.516728 19153 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:03:15.520260 19221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:03:15.522217 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 19153
I20260806 17:03:15.522409 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:03:15.522713 9661 external_mini_cluster.cc:1428] Setting key 712bb7f6a2d3e60758edc78ad5d567ae
I20260806 17:03:15.527434 19221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4: Bootstrap starting.
I20260806 17:03:15.530251 19221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4: Neither blocks nor log segments found. Creating new log.
I20260806 17:03:15.531204 19221 log.cc:824] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4: Log is configured to *not* fsync() on all Append() calls
I20260806 17:03:15.533687 19221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4: No bootstrap required, opened a new log
I20260806 17:03:15.534530 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:15.524413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48242 (local address 127.9.111.126:44459)
0806 17:03:15.525143 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:15.525166 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:15.525212 (+ 46us) server_negotiation.cc:407] Connection header received
0806 17:03:15.526243 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:15.526268 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:15.526790 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:15.527162 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:15.527613 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:15.529084 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:15.530153 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:15.530891 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:15.531075 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:15.532127 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:03:15.532135 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:03:15.532156 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:03:15.532586 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:03:15.532854 (+ 268us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:03:15.532909 (+ 55us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:03:15.533076 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:15.533315 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 17:03:15.533657 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":118,"threads_started":1}
I20260806 17:03:15.537951 19221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58cde0796ef74d9488ba4a5a82f2d4c4" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44459 } }
I20260806 17:03:15.538208 19221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:03:15.538285 19221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58cde0796ef74d9488ba4a5a82f2d4c4, State: Initialized, Role: FOLLOWER
I20260806 17:03:15.538702 19221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58cde0796ef74d9488ba4a5a82f2d4c4" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44459 } }
I20260806 17:03:15.538822 19221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:03:15.538913 19221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:03:15.539011 19221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:03:15.540014 19221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58cde0796ef74d9488ba4a5a82f2d4c4" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44459 } }
I20260806 17:03:15.540378 19221 leader_election.cc:302] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58cde0796ef74d9488ba4a5a82f2d4c4; no voters:
I20260806 17:03:15.540704 19221 leader_election.cc:288] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:03:15.540856 19227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:03:15.541126 19227 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4 [term 1 LEADER]: Becoming Leader. State: Replica: 58cde0796ef74d9488ba4a5a82f2d4c4, State: Running, Role: LEADER
I20260806 17:03:15.541533 19227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58cde0796ef74d9488ba4a5a82f2d4c4" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44459 } }
I20260806 17:03:15.542280 19221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:03:15.543215 19228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58cde0796ef74d9488ba4a5a82f2d4c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58cde0796ef74d9488ba4a5a82f2d4c4" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44459 } } }
I20260806 17:03:15.543346 19228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4 [sys.catalog]: This master's current role is: LEADER
I20260806 17:03:15.543809 19229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58cde0796ef74d9488ba4a5a82f2d4c4. Latest consensus state: current_term: 1 leader_uuid: "58cde0796ef74d9488ba4a5a82f2d4c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58cde0796ef74d9488ba4a5a82f2d4c4" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 44459 } } }
I20260806 17:03:15.543910 19229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4 [sys.catalog]: This master's current role is: LEADER
I20260806 17:03:15.544818 19238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:03:15.548622 19238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:03:15.555124 19238 catalog_manager.cc:1347] Generated new cluster ID: e97806ed91334424b7a0d9c9354221af
I20260806 17:03:15.555219 19238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:03:15.573791 19238 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:03:15.574863 19238 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:03:15.589661 19238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58cde0796ef74d9488ba4a5a82f2d4c4: Generated new TSK 0
I20260806 17:03:15.590659 19238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:03:15.604031 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:44459
--builtin_ntp_servers=127.9.111.84:33227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 17:03:15.729367 19248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:03:15.729667 19248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:03:15.729705 19248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:03:15.733946 19248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:03:15.734023 19248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:03:15.734110 19248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:03:15.739089 19248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:33227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:44459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:03:15.740345 19248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:03:15.741947 19248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:03:15.756356 19256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:15.756350 19253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:03:15.757041 19248 server_base.cc:1034] running on GCE node
W20260806 17:03:15.757050 19254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:03:15.757784 19248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:03:15.758692 19248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:03:15.759912 19248 hybrid_clock.cc:648] HybridClock initialized: now 1786035795759871 us; error 51 us; skew 500 ppm
I20260806 17:03:15.762596 19248 webserver.cc:460] Webserver started at http://127.9.111.65:39847/ using document root <none> and password file <none>
I20260806 17:03:15.763314 19248 fs_manager.cc:359] Metadata directory not provided
I20260806 17:03:15.763408 19248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:03:15.763598 19248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:03:15.765791 19248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "8c29e4a0be1e46ef8b07fe0edd073cdf"
format_stamp: "Formatted at 2026-08-06 17:03:15 on dist-test-slave-69rg"
server_key: "c1b81b118ad5614b687c8caf33d1ae8e"
server_key_iv: "c1299a9b95dc513d7d96e4ba76a25a86"
server_key_version: "encryptionkey@0"
I20260806 17:03:15.766471 19248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "8c29e4a0be1e46ef8b07fe0edd073cdf"
format_stamp: "Formatted at 2026-08-06 17:03:15 on dist-test-slave-69rg"
server_key: "c1b81b118ad5614b687c8caf33d1ae8e"
server_key_iv: "c1299a9b95dc513d7d96e4ba76a25a86"
server_key_version: "encryptionkey@0"
I20260806 17:03:15.770862 19248 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:03:15.773778 19263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:15.775223 19248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:03:15.775359 19248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "8c29e4a0be1e46ef8b07fe0edd073cdf"
format_stamp: "Formatted at 2026-08-06 17:03:15 on dist-test-slave-69rg"
server_key: "c1b81b118ad5614b687c8caf33d1ae8e"
server_key_iv: "c1299a9b95dc513d7d96e4ba76a25a86"
server_key_version: "encryptionkey@0"
I20260806 17:03:15.775498 19248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:03:15.788946 19248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:03:15.789808 19248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:03:15.790081 19248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:03:15.790767 19248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:03:15.791870 19248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:03:15.791924 19248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:15.791965 19248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:03:15.791980 19248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:15.804090 19248 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:36225
I20260806 17:03:15.804188 19383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:36225 every 8 connection(s)
I20260806 17:03:15.805151 19248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:03:15.813033 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 19248
I20260806 17:03:15.813189 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:03:15.813568 9661 external_mini_cluster.cc:1428] Setting key eb92313ba0ff4b614256a68519fb84a4
I20260806 17:03:15.816031 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:15.807379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:53345 (local address 127.9.111.126:44459)
0806 17:03:15.807541 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:15.807545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:15.808618 (+ 1073us) server_negotiation.cc:407] Connection header received
0806 17:03:15.810093 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:15.810096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:15.810150 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:15.810234 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:15.811296 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:15.811856 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:15.813458 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:15.813759 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:15.813915 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:15.815311 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:03:15.815317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:03:15.815320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:03:15.815337 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:03:15.815368 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:03:15.815383 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:03:15.815470 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:15.815810 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 17:03:15.815879 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 17:03:15.816593 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:44459
--builtin_ntp_servers=127.9.111.84:33227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 17:03:15.817075 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:15.807771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:44459 (local address 127.9.111.65:53345)
0806 17:03:15.808464 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:15.808507 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:15.809881 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:15.810447 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:15.810454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:15.810543 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:15.811129 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:15.811143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:15.811993 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:15.811996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:15.813320 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:15.813333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:15.813966 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:15.813971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:15.814203 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:15.814925 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:03:15.814928 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:15.814986 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:03:15.815074 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:03:15.815565 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:03:15.815576 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:03:15.815582 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 17:03:15.815748 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 17:03:15.816030 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":135,"threads_started":1}
I20260806 17:03:15.818598 19384 heartbeater.cc:347] Connected to a master server at 127.9.111.126:44459
I20260806 17:03:15.818936 19384 heartbeater.cc:464] Registering TS with master...
I20260806 17:03:15.819592 19384 heartbeater.cc:511] Master 127.9.111.126:44459 requested a full tablet report, sending...
I20260806 17:03:15.822057 19184 ts_manager.cc:194] Registered new tserver with Master: 8c29e4a0be1e46ef8b07fe0edd073cdf (127.9.111.65:36225)
I20260806 17:03:15.823311 19184 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.111.65:53345
W20260806 17:03:15.954743 19388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:03:15.955057 19388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:03:15.955098 19388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:03:15.959820 19388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:03:15.959960 19388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:03:15.960227 19388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:03:15.966708 19388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:33227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:44459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:03:15.967941 19388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:03:15.969422 19388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:03:15.980535 19394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:15.980535 19396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:15.980531 19393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:03:15.980942 19388 server_base.cc:1034] running on GCE node
I20260806 17:03:15.981437 19388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:03:15.982252 19388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:03:15.983457 19388 hybrid_clock.cc:648] HybridClock initialized: now 1786035795983435 us; error 39 us; skew 500 ppm
I20260806 17:03:15.990168 19388 webserver.cc:460] Webserver started at http://127.9.111.66:34593/ using document root <none> and password file <none>
I20260806 17:03:15.990871 19388 fs_manager.cc:359] Metadata directory not provided
I20260806 17:03:15.990981 19388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:03:15.991247 19388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:03:15.993449 19388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "ad1bad411b634284957832ecdd8feee4"
format_stamp: "Formatted at 2026-08-06 17:03:15 on dist-test-slave-69rg"
server_key: "0c2a261faac1181f3babf96a0ef05900"
server_key_iv: "88cc7b58cfb5b7bd487df4ba90a18abc"
server_key_version: "encryptionkey@0"
I20260806 17:03:15.994096 19388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "ad1bad411b634284957832ecdd8feee4"
format_stamp: "Formatted at 2026-08-06 17:03:15 on dist-test-slave-69rg"
server_key: "0c2a261faac1181f3babf96a0ef05900"
server_key_iv: "88cc7b58cfb5b7bd487df4ba90a18abc"
server_key_version: "encryptionkey@0"
I20260806 17:03:15.998028 19388 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 17:03:16.001070 19402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:16.002665 19388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:03:16.002830 19388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "ad1bad411b634284957832ecdd8feee4"
format_stamp: "Formatted at 2026-08-06 17:03:15 on dist-test-slave-69rg"
server_key: "0c2a261faac1181f3babf96a0ef05900"
server_key_iv: "88cc7b58cfb5b7bd487df4ba90a18abc"
server_key_version: "encryptionkey@0"
I20260806 17:03:16.002969 19388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:03:16.021579 19388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:03:16.022488 19388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:03:16.022734 19388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:03:16.023597 19388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:03:16.024695 19388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:03:16.024777 19388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:16.024845 19388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:03:16.024899 19388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:16.036338 19388 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:44837
I20260806 17:03:16.036417 19519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:44837 every 8 connection(s)
I20260806 17:03:16.037393 19388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:03:16.047498 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 19388
I20260806 17:03:16.047610 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:03:16.047904 9661 external_mini_cluster.cc:1428] Setting key 26000c3580eb32351181d34024da732a
I20260806 17:03:16.050014 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:16.039684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:58943 (local address 127.9.111.126:44459)
0806 17:03:16.039867 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:16.039872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:16.041037 (+ 1165us) server_negotiation.cc:407] Connection header received
0806 17:03:16.043124 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:16.043128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:16.043207 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:16.043288 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:16.044756 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:16.045541 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:16.046832 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:16.047070 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:16.047266 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:16.049015 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:03:16.049019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:03:16.049022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:03:16.049035 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:03:16.049058 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:03:16.049068 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:03:16.049308 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:16.049730 (+ 422us) server_negotiation.cc:299] Negotiation successful
0806 17:03:16.049857 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 17:03:16.050689 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:44459
--builtin_ntp_servers=127.9.111.84:33227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 17:03:16.051198 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:16.040121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:44459 (local address 127.9.111.66:58943)
0806 17:03:16.040872 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:16.040917 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:16.042412 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:16.043857 (+ 1445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:16.043869 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:16.043990 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:16.044603 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:16.044617 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:16.045680 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:16.045683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:16.046688 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:16.046701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:16.047471 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:16.047477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:16.047887 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:16.048669 (+ 782us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:03:16.048671 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:16.048734 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:03:16.048838 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:03:16.049341 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:03:16.049350 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:03:16.049356 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 17:03:16.049570 (+ 214us) client_negotiation.cc:240] Negotiation successful
0806 17:03:16.050158 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":139,"threads_started":1}
I20260806 17:03:16.052489 19520 heartbeater.cc:347] Connected to a master server at 127.9.111.126:44459
I20260806 17:03:16.052781 19520 heartbeater.cc:464] Registering TS with master...
I20260806 17:03:16.053400 19520 heartbeater.cc:511] Master 127.9.111.126:44459 requested a full tablet report, sending...
I20260806 17:03:16.054670 19184 ts_manager.cc:194] Registered new tserver with Master: ad1bad411b634284957832ecdd8feee4 (127.9.111.66:44837)
I20260806 17:03:16.055604 19184 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.111.66:58943
W20260806 17:03:16.183764 19523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:03:16.184070 19523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:03:16.184113 19523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:03:16.188375 19523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:03:16.188459 19523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:03:16.188517 19523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:03:16.193481 19523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:33227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:44459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:03:16.194972 19523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:03:16.196635 19523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:03:16.209110 19529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:03:16.209353 19523 server_base.cc:1034] running on GCE node
W20260806 17:03:16.209126 19531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:16.209267 19528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:03:16.210075 19523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:03:16.210829 19523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:03:16.212069 19523 hybrid_clock.cc:648] HybridClock initialized: now 1786035796212033 us; error 55 us; skew 500 ppm
I20260806 17:03:16.214416 19523 webserver.cc:460] Webserver started at http://127.9.111.67:36697/ using document root <none> and password file <none>
I20260806 17:03:16.215047 19523 fs_manager.cc:359] Metadata directory not provided
I20260806 17:03:16.215188 19523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:03:16.215435 19523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:03:16.217653 19523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "fa315176f46144eaabfd422147de2ee6"
format_stamp: "Formatted at 2026-08-06 17:03:16 on dist-test-slave-69rg"
server_key: "6fb3e26c9e871fc28acd29ece963b88a"
server_key_iv: "2152ed8222316ca2fb405c03452bbd9d"
server_key_version: "encryptionkey@0"
I20260806 17:03:16.218343 19523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "fa315176f46144eaabfd422147de2ee6"
format_stamp: "Formatted at 2026-08-06 17:03:16 on dist-test-slave-69rg"
server_key: "6fb3e26c9e871fc28acd29ece963b88a"
server_key_iv: "2152ed8222316ca2fb405c03452bbd9d"
server_key_version: "encryptionkey@0"
I20260806 17:03:16.223129 19523 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:03:16.226106 19537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:16.227849 19523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:03:16.228022 19523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "fa315176f46144eaabfd422147de2ee6"
format_stamp: "Formatted at 2026-08-06 17:03:16 on dist-test-slave-69rg"
server_key: "6fb3e26c9e871fc28acd29ece963b88a"
server_key_iv: "2152ed8222316ca2fb405c03452bbd9d"
server_key_version: "encryptionkey@0"
I20260806 17:03:16.228173 19523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:03:16.239732 19523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:03:16.240722 19523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:03:16.240934 19523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:03:16.241602 19523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:03:16.243214 19523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:03:16.243310 19523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:16.243396 19523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:03:16.243455 19523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:16.253765 19523 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:42585
I20260806 17:03:16.253880 19658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:42585 every 8 connection(s)
I20260806 17:03:16.254803 19523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:03:16.260823 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 19523
I20260806 17:03:16.260969 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:03:16.261276 9661 external_mini_cluster.cc:1428] Setting key 4599c846b4ad35e8a0e703c6c34992a0
I20260806 17:03:16.265450 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:16.256861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:43703 (local address 127.9.111.126:44459)
0806 17:03:16.257064 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:16.257068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:16.258278 (+ 1210us) server_negotiation.cc:407] Connection header received
0806 17:03:16.259718 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:16.259721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:16.259777 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:16.259860 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:16.261001 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:16.261660 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:16.263337 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:16.263569 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:16.263721 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:16.264931 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:03:16.264934 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:03:16.264936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:03:16.264948 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:03:16.264967 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:03:16.264975 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:03:16.265052 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:16.265229 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 17:03:16.265295 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260806 17:03:16.266522 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:16.257299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:44459 (local address 127.9.111.67:43703)
0806 17:03:16.258078 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:16.258165 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:16.259477 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:16.260028 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:16.260034 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:16.260151 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:16.260806 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:16.260827 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:16.261818 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:16.261821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:16.263148 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:16.263161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:16.263759 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:16.263763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:16.263966 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:16.264619 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:03:16.264621 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:16.264673 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:03:16.264745 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:03:16.265068 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:03:16.265077 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:03:16.265083 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 17:03:16.265230 (+ 147us) client_negotiation.cc:240] Negotiation successful
0806 17:03:16.265530 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":129,"threads_started":1}
I20260806 17:03:16.267747 19659 heartbeater.cc:347] Connected to a master server at 127.9.111.126:44459
I20260806 17:03:16.267998 19659 heartbeater.cc:464] Registering TS with master...
I20260806 17:03:16.268517 19659 heartbeater.cc:511] Master 127.9.111.126:44459 requested a full tablet report, sending...
I20260806 17:03:16.269945 19184 ts_manager.cc:194] Registered new tserver with Master: fa315176f46144eaabfd422147de2ee6 (127.9.111.67:42585)
I20260806 17:03:16.270632 19184 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.111.67:43703
I20260806 17:03:16.276000 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 17:03:16.279325 19225 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:16.278169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.9.111.126:44459)
0806 17:03:16.278330 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:16.278338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:16.278514 (+ 176us) server_negotiation.cc:407] Connection header received
0806 17:03:16.278589 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:16.278591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:16.278689 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:16.278782 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:16.279167 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48258: BlockingRecv error: recv got EOF from 127.0.0.1:48258 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 17:03:16.279448 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 19248
I20260806 17:03:16.286072 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 19388
I20260806 17:03:16.291854 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 19523
I20260806 17:03:16.297637 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 19153
2026-08-06T17:03:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1039 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T17:03:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:03:16Z Disabled control of system clock
I20260806 17:03:16.330533 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:36649
--webserver_interface=127.9.111.126
--webserver_port=0
--builtin_ntp_servers=127.9.111.84:45345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:36649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 17:03:16.451773 19670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:03:16.452085 19670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:03:16.452124 19670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:03:16.457188 19670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:03:16.457288 19670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:03:16.457305 19670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:03:16.457324 19670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:03:16.457340 19670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:03:16.462672 19670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:45345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:36649
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:36649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:03:16.463945 19670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:03:16.465369 19670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:03:16.475024 19676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:16.475024 19678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:16.475069 19675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:03:16.475476 19670 server_base.cc:1034] running on GCE node
I20260806 17:03:16.476176 19670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:03:16.477456 19670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:03:16.478704 19670 hybrid_clock.cc:648] HybridClock initialized: now 1786035796478683 us; error 36 us; skew 500 ppm
I20260806 17:03:16.480762 19670 webserver.cc:460] Webserver started at http://127.9.111.126:35487/ using document root <none> and password file <none>
I20260806 17:03:16.481324 19670 fs_manager.cc:359] Metadata directory not provided
I20260806 17:03:16.481391 19670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:03:16.481566 19670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:03:16.483726 19670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "1404a7e0838e49e3bd23980d6332c1a1"
format_stamp: "Formatted at 2026-08-06 17:03:16 on dist-test-slave-69rg"
server_key: "cf06b2e7b9ec06bbf67830c26a2f81e2"
server_key_iv: "5350e75573da94e70a3ddf0bf442afea"
server_key_version: "encryptionkey@0"
I20260806 17:03:16.484248 19670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "1404a7e0838e49e3bd23980d6332c1a1"
format_stamp: "Formatted at 2026-08-06 17:03:16 on dist-test-slave-69rg"
server_key: "cf06b2e7b9ec06bbf67830c26a2f81e2"
server_key_iv: "5350e75573da94e70a3ddf0bf442afea"
server_key_version: "encryptionkey@0"
I20260806 17:03:16.488068 19670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 17:03:16.490811 19684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:16.492300 19670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 17:03:16.492435 19670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "1404a7e0838e49e3bd23980d6332c1a1"
format_stamp: "Formatted at 2026-08-06 17:03:16 on dist-test-slave-69rg"
server_key: "cf06b2e7b9ec06bbf67830c26a2f81e2"
server_key_iv: "5350e75573da94e70a3ddf0bf442afea"
server_key_version: "encryptionkey@0"
I20260806 17:03:16.492547 19670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:03:16.497334 19670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:03:16.498031 19670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:03:16.498208 19670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:03:16.505546 19670 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:36649
I20260806 17:03:16.505585 19739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:36649 every 8 connection(s)
I20260806 17:03:16.506903 19670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:03:16.508033 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 19670
I20260806 17:03:16.508129 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:03:16.508551 9661 external_mini_cluster.cc:1428] Setting key e52c98cd93c62c91dc521ae84005abc8
I20260806 17:03:16.510754 19740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:03:16.514904 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:16.510058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39834 (local address 127.9.111.126:36649)
0806 17:03:16.511142 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:16.511159 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:16.511196 (+ 37us) server_negotiation.cc:407] Connection header received
0806 17:03:16.511863 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:16.511878 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:16.512250 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:16.512543 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:16.513465 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:03:16.513469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:03:16.513480 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:03:16.513488 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:03:16.513540 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:03:16.513555 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:03:16.513685 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:16.513794 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 17:03:16.514152 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":592,"threads_started":1}
I20260806 17:03:16.517458 19740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1: Bootstrap starting.
I20260806 17:03:16.519778 19740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1: Neither blocks nor log segments found. Creating new log.
I20260806 17:03:16.520545 19740 log.cc:824] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1: Log is configured to *not* fsync() on all Append() calls
I20260806 17:03:16.522643 19740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1: No bootstrap required, opened a new log
I20260806 17:03:16.526429 19740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1404a7e0838e49e3bd23980d6332c1a1" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36649 } }
I20260806 17:03:16.526690 19740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:03:16.526785 19740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1404a7e0838e49e3bd23980d6332c1a1, State: Initialized, Role: FOLLOWER
I20260806 17:03:16.527273 19740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1404a7e0838e49e3bd23980d6332c1a1" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36649 } }
I20260806 17:03:16.527427 19740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:03:16.527518 19740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:03:16.527617 19740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:03:16.528625 19740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1404a7e0838e49e3bd23980d6332c1a1" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36649 } }
I20260806 17:03:16.528985 19740 leader_election.cc:302] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1404a7e0838e49e3bd23980d6332c1a1; no voters:
I20260806 17:03:16.529320 19740 leader_election.cc:288] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:03:16.529505 19745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:03:16.529800 19745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1 [term 1 LEADER]: Becoming Leader. State: Replica: 1404a7e0838e49e3bd23980d6332c1a1, State: Running, Role: LEADER
I20260806 17:03:16.530166 19745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1404a7e0838e49e3bd23980d6332c1a1" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36649 } }
I20260806 17:03:16.530666 19740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:03:16.531944 19747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1404a7e0838e49e3bd23980d6332c1a1. Latest consensus state: current_term: 1 leader_uuid: "1404a7e0838e49e3bd23980d6332c1a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1404a7e0838e49e3bd23980d6332c1a1" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36649 } } }
I20260806 17:03:16.532089 19747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1 [sys.catalog]: This master's current role is: LEADER
I20260806 17:03:16.532325 19746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1404a7e0838e49e3bd23980d6332c1a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1404a7e0838e49e3bd23980d6332c1a1" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36649 } } }
I20260806 17:03:16.532482 19746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1 [sys.catalog]: This master's current role is: LEADER
I20260806 17:03:16.532506 19754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:03:16.535998 19754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:03:16.541615 19754 catalog_manager.cc:1347] Generated new cluster ID: ad3a6f23f251485e899aeac14aa57613
I20260806 17:03:16.541700 19754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:03:16.562983 19754 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:03:16.563376 19754 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:03:16.568964 19754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1404a7e0838e49e3bd23980d6332c1a1: Generated new TSK 0
I20260806 17:03:16.569796 19754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:03:16.582722 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:36649
--builtin_ntp_servers=127.9.111.84:45345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 17:03:16.708487 19764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:03:16.708806 19764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:03:16.708849 19764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:03:16.713065 19764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:03:16.713142 19764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:03:16.713207 19764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:03:16.718302 19764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:45345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:36649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:03:16.719568 19764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:03:16.721057 19764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:03:16.731508 19771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:16.731491 19773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:16.731495 19770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:03:16.731585 19764 server_base.cc:1034] running on GCE node
I20260806 17:03:16.732340 19764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:03:16.733057 19764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:03:16.734284 19764 hybrid_clock.cc:648] HybridClock initialized: now 1786035796734248 us; error 50 us; skew 500 ppm
I20260806 17:03:16.736465 19764 webserver.cc:460] Webserver started at http://127.9.111.65:46847/ using document root <none> and password file <none>
I20260806 17:03:16.737106 19764 fs_manager.cc:359] Metadata directory not provided
I20260806 17:03:16.737195 19764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:03:16.737596 19764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:03:16.740072 19764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "e46e009cbfa04f8ab99b4ba1889ae025"
format_stamp: "Formatted at 2026-08-06 17:03:16 on dist-test-slave-69rg"
server_key: "f5fdc4becacb9d1d22dd3f0e244532c2"
server_key_iv: "2dc363e563f93c078002cb578544c165"
server_key_version: "encryptionkey@0"
I20260806 17:03:16.740725 19764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "e46e009cbfa04f8ab99b4ba1889ae025"
format_stamp: "Formatted at 2026-08-06 17:03:16 on dist-test-slave-69rg"
server_key: "f5fdc4becacb9d1d22dd3f0e244532c2"
server_key_iv: "2dc363e563f93c078002cb578544c165"
server_key_version: "encryptionkey@0"
I20260806 17:03:16.745069 19764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260806 17:03:16.748322 19779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:16.749752 19764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 17:03:16.749938 19764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "e46e009cbfa04f8ab99b4ba1889ae025"
format_stamp: "Formatted at 2026-08-06 17:03:16 on dist-test-slave-69rg"
server_key: "f5fdc4becacb9d1d22dd3f0e244532c2"
server_key_iv: "2dc363e563f93c078002cb578544c165"
server_key_version: "encryptionkey@0"
I20260806 17:03:16.750087 19764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:03:16.754985 19764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:03:16.755707 19764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:03:16.755920 19764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:03:16.756544 19764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:03:16.757923 19764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:03:16.758002 19764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:16.758072 19764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:03:16.758142 19764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:16.767829 19764 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:39669
I20260806 17:03:16.767865 19896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:39669 every 8 connection(s)
I20260806 17:03:16.768805 19764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:03:16.771317 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 19764
I20260806 17:03:16.771463 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:03:16.771798 9661 external_mini_cluster.cc:1428] Setting key dfd7ee94e0e1b73708f715240e6f18e8
I20260806 17:03:16.773968 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:36649
--builtin_ntp_servers=127.9.111.84:45345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 17:03:16.776269 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:16.771010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:52323 (local address 127.9.111.126:36649)
0806 17:03:16.771157 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:16.771161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:16.772426 (+ 1265us) server_negotiation.cc:407] Connection header received
0806 17:03:16.773953 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:16.773957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:16.774004 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:16.774098 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:16.775388 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:03:16.775394 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:03:16.775400 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:03:16.775420 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:03:16.775448 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:03:16.775462 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:03:16.775587 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:16.776062 (+ 475us) server_negotiation.cc:299] Negotiation successful
0806 17:03:16.776114 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 17:03:16.777299 19899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:16.771350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:36649 (local address 127.9.111.65:52323)
0806 17:03:16.772239 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:16.772302 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:16.773675 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:16.774320 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:16.774328 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:16.774420 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:16.775078 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:03:16.775080 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:16.775124 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:03:16.775198 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:03:16.775755 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:03:16.775769 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:03:16.775774 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 17:03:16.776116 (+ 342us) client_negotiation.cc:240] Negotiation successful
0806 17:03:16.776499 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":153,"threads_started":1}
I20260806 17:03:16.778785 19897 heartbeater.cc:347] Connected to a master server at 127.9.111.126:36649
I20260806 17:03:16.779080 19897 heartbeater.cc:464] Registering TS with master...
I20260806 17:03:16.779579 19897 heartbeater.cc:511] Master 127.9.111.126:36649 requested a full tablet report, sending...
I20260806 17:03:16.781395 19701 ts_manager.cc:194] Registered new tserver with Master: e46e009cbfa04f8ab99b4ba1889ae025 (127.9.111.65:39669)
W20260806 17:03:16.893386 19900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:03:16.893850 19900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:03:16.893963 19900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:03:16.898751 19900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:03:16.898882 19900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:03:16.898988 19900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:03:16.904394 19900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:45345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:36649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:03:16.905580 19900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:03:16.907097 19900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:03:16.918833 19909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:16.918840 19905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:16.919490 19907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:03:16.919638 19900 server_base.cc:1034] running on GCE node
I20260806 17:03:16.920222 19900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:03:16.921054 19900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:03:16.924609 19900 hybrid_clock.cc:648] HybridClock initialized: now 1786035796924574 us; error 46 us; skew 500 ppm
I20260806 17:03:16.927090 19900 webserver.cc:460] Webserver started at http://127.9.111.66:43149/ using document root <none> and password file <none>
I20260806 17:03:16.927767 19900 fs_manager.cc:359] Metadata directory not provided
I20260806 17:03:16.927872 19900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:03:16.928129 19900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:03:16.930425 19900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "38deb1cad0d14474ba4628a690221c32"
format_stamp: "Formatted at 2026-08-06 17:03:16 on dist-test-slave-69rg"
server_key: "e4d318a2db4943f80c725d85e6d8698c"
server_key_iv: "0431703852c03626ef136576bc4e67c8"
server_key_version: "encryptionkey@0"
I20260806 17:03:16.931058 19900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "38deb1cad0d14474ba4628a690221c32"
format_stamp: "Formatted at 2026-08-06 17:03:16 on dist-test-slave-69rg"
server_key: "e4d318a2db4943f80c725d85e6d8698c"
server_key_iv: "0431703852c03626ef136576bc4e67c8"
server_key_version: "encryptionkey@0"
I20260806 17:03:16.935434 19900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:03:16.938364 19916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:16.939761 19900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:03:16.939941 19900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "38deb1cad0d14474ba4628a690221c32"
format_stamp: "Formatted at 2026-08-06 17:03:16 on dist-test-slave-69rg"
server_key: "e4d318a2db4943f80c725d85e6d8698c"
server_key_iv: "0431703852c03626ef136576bc4e67c8"
server_key_version: "encryptionkey@0"
I20260806 17:03:16.940097 19900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:03:16.945478 19900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:03:16.946223 19900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:03:16.946440 19900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:03:16.947023 19900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:03:16.947992 19900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:03:16.948076 19900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:16.948171 19900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:03:16.948222 19900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:16.958063 19900 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:44677
I20260806 17:03:16.958087 20033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:44677 every 8 connection(s)
I20260806 17:03:16.959061 19900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:03:16.963531 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 19900
I20260806 17:03:16.963650 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:03:16.963970 9661 external_mini_cluster.cc:1428] Setting key cef93288f16369d2265877afccf243a6
I20260806 17:03:16.965909 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:16.961023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:37049 (local address 127.9.111.126:36649)
0806 17:03:16.961175 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:16.961180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:16.962404 (+ 1224us) server_negotiation.cc:407] Connection header received
0806 17:03:16.963879 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:16.963883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:16.963938 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:16.964052 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:16.965375 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:03:16.965378 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:03:16.965380 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:03:16.965399 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:03:16.965427 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:03:16.965435 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:03:16.965536 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:16.965686 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 17:03:16.965733 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 17:03:16.966574 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:36649
--builtin_ntp_servers=127.9.111.84:45345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 17:03:16.967344 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:16.961384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:36649 (local address 127.9.111.66:37049)
0806 17:03:16.962197 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:16.962244 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:16.963569 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:16.964247 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:16.964257 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:16.964392 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:16.965051 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:03:16.965053 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:16.965096 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:03:16.965171 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:03:16.965527 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:03:16.965537 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:03:16.965544 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 17:03:16.965739 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 17:03:16.966116 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":111,"threads_started":1}
I20260806 17:03:16.968703 20034 heartbeater.cc:347] Connected to a master server at 127.9.111.126:36649
I20260806 17:03:16.968981 20034 heartbeater.cc:464] Registering TS with master...
I20260806 17:03:16.969496 20034 heartbeater.cc:511] Master 127.9.111.126:36649 requested a full tablet report, sending...
I20260806 17:03:16.970788 19701 ts_manager.cc:194] Registered new tserver with Master: 38deb1cad0d14474ba4628a690221c32 (127.9.111.66:44677)
W20260806 17:03:17.087955 20037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:03:17.088263 20037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:03:17.088305 20037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:03:17.092494 20037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:03:17.092730 20037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:03:17.092849 20037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:03:17.098732 20037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:45345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:36649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:03:17.099923 20037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:03:17.101353 20037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:03:17.111404 20042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:17.111413 20043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:17.111452 20045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:03:17.111867 20037 server_base.cc:1034] running on GCE node
I20260806 17:03:17.112265 20037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:03:17.112972 20037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:03:17.114142 20037 hybrid_clock.cc:648] HybridClock initialized: now 1786035797114124 us; error 29 us; skew 500 ppm
I20260806 17:03:17.116078 20037 webserver.cc:460] Webserver started at http://127.9.111.67:46179/ using document root <none> and password file <none>
I20260806 17:03:17.116686 20037 fs_manager.cc:359] Metadata directory not provided
I20260806 17:03:17.116787 20037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:03:17.117053 20037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:03:17.119578 20037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "bc3de162cb2445e3bccfd992b2ebcf21"
format_stamp: "Formatted at 2026-08-06 17:03:17 on dist-test-slave-69rg"
server_key: "4684dc0f53f69a55a71f4725197a8f39"
server_key_iv: "3c8376d70740c4e394f4886cd0cd790d"
server_key_version: "encryptionkey@0"
I20260806 17:03:17.120229 20037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "bc3de162cb2445e3bccfd992b2ebcf21"
format_stamp: "Formatted at 2026-08-06 17:03:17 on dist-test-slave-69rg"
server_key: "4684dc0f53f69a55a71f4725197a8f39"
server_key_iv: "3c8376d70740c4e394f4886cd0cd790d"
server_key_version: "encryptionkey@0"
I20260806 17:03:17.125208 20037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 17:03:17.129014 20051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:17.130844 20037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:03:17.131022 20037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "bc3de162cb2445e3bccfd992b2ebcf21"
format_stamp: "Formatted at 2026-08-06 17:03:17 on dist-test-slave-69rg"
server_key: "4684dc0f53f69a55a71f4725197a8f39"
server_key_iv: "3c8376d70740c4e394f4886cd0cd790d"
server_key_version: "encryptionkey@0"
I20260806 17:03:17.131206 20037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:03:17.136282 20037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:03:17.136967 20037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:03:17.137171 20037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:03:17.137763 20037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:03:17.138809 20037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:03:17.138885 20037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:17.138957 20037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:03:17.138999 20037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:17.149031 20037 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:46493
I20260806 17:03:17.149053 20168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:46493 every 8 connection(s)
I20260806 17:03:17.150076 20037 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:03:17.155273 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 20037
I20260806 17:03:17.155402 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:03:17.155733 9661 external_mini_cluster.cc:1428] Setting key 6caef62579dcb07f8d356d0f3350a513
I20260806 17:03:17.157088 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:17.152168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:45805 (local address 127.9.111.126:36649)
0806 17:03:17.152370 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:17.152376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:17.153560 (+ 1184us) server_negotiation.cc:407] Connection header received
0806 17:03:17.155046 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:17.155049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:17.155091 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:17.155207 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:17.156500 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:03:17.156506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:03:17.156509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:03:17.156523 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:03:17.156549 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:03:17.156559 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:03:17.156646 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:17.156905 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 17:03:17.156952 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 17:03:17.158347 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:17.152612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:36649 (local address 127.9.111.67:45805)
0806 17:03:17.153397 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:17.153440 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:17.154797 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:17.155397 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:17.155405 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:17.155519 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:17.156242 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:03:17.156244 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:17.156286 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:03:17.156359 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:03:17.156711 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:03:17.156722 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:03:17.156731 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 17:03:17.156952 (+ 221us) client_negotiation.cc:240] Negotiation successful
0806 17:03:17.157307 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":157,"threads_started":1}
I20260806 17:03:17.159437 20169 heartbeater.cc:347] Connected to a master server at 127.9.111.126:36649
I20260806 17:03:17.159643 20169 heartbeater.cc:464] Registering TS with master...
I20260806 17:03:17.160014 20169 heartbeater.cc:511] Master 127.9.111.126:36649 requested a full tablet report, sending...
I20260806 17:03:17.160943 19701 ts_manager.cc:194] Registered new tserver with Master: bc3de162cb2445e3bccfd992b2ebcf21 (127.9.111.67:46493)
I20260806 17:03:17.170646 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 17:03:17.173784 19743 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:03:17.172866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.9.111.126:36649)
0806 17:03:17.172989 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:17.172993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:17.173070 (+ 77us) server_negotiation.cc:407] Connection header received
0806 17:03:17.173222 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:17.173225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:17.173260 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:17.173338 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:17.173618 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39844: BlockingRecv error: recv got EOF from 127.0.0.1:39844 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 17:03:17.174095 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 19764
I20260806 17:03:17.180536 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 19900
I20260806 17:03:17.186529 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 20037
I20260806 17:03:17.192479 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 19670
2026-08-06T17:03:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (894 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:03:17 dist-test-slave-69rg krb5kdc[20180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:03:17 dist-test-slave-69rg krb5kdc[20180](info): set up 2 sockets
Aug 06 17:03:17 dist-test-slave-69rg krb5kdc[20180](info): commencing operation
krb5kdc: starting...
W20260806 17:03:18.274572 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:03:18 dist-test-slave-69rg krb5kdc[20180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:03:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:03:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:03:18.411656 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:43057
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:43057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:03:18.526612 20194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:03:18.526971 20194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:03:18.527050 20194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:03:18.531178 20194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:03:18.531293 20194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:03:18.531332 20194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:03:18.531385 20194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:03:18.531422 20194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:03:18.536988 20194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:43057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:43057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:03:18.538298 20194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:03:18.539395 20194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:03:18.549068 20200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:18.549077 20201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:03:18.549127 20194 server_base.cc:1034] running on GCE node
W20260806 17:03:18.549070 20203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:03:18.549995 20194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:03:18.551165 20194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:03:18.552337 20194 hybrid_clock.cc:648] HybridClock initialized: now 1786035798552306 us; error 41 us; skew 500 ppm
Aug 06 17:03:18 dist-test-slave-69rg krb5kdc[20180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035798, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:03:18.555243 20194 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:03:18.556111 20194 webserver.cc:460] Webserver started at http://127.9.111.126:35503/ using document root <none> and password file <none>
I20260806 17:03:18.556677 20194 fs_manager.cc:359] Metadata directory not provided
I20260806 17:03:18.556772 20194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:03:18.557009 20194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:03:18.559226 20194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "2ac067dab29a44f5b6945b68ef92fb31"
format_stamp: "Formatted at 2026-08-06 17:03:18 on dist-test-slave-69rg"
server_key: "e16833e5ad20602ba295e1be4ede598b"
server_key_iv: "4cbe76820e071e9f62952673f5f4fb5a"
server_key_version: "encryptionkey@0"
I20260806 17:03:18.559823 20194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "2ac067dab29a44f5b6945b68ef92fb31"
format_stamp: "Formatted at 2026-08-06 17:03:18 on dist-test-slave-69rg"
server_key: "e16833e5ad20602ba295e1be4ede598b"
server_key_iv: "4cbe76820e071e9f62952673f5f4fb5a"
server_key_version: "encryptionkey@0"
I20260806 17:03:18.563767 20194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 17:03:18.566298 20210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:18.567593 20194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:03:18.567734 20194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "2ac067dab29a44f5b6945b68ef92fb31"
format_stamp: "Formatted at 2026-08-06 17:03:18 on dist-test-slave-69rg"
server_key: "e16833e5ad20602ba295e1be4ede598b"
server_key_iv: "4cbe76820e071e9f62952673f5f4fb5a"
server_key_version: "encryptionkey@0"
I20260806 17:03:18.567852 20194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:03:18.596020 20194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:03:18.599447 20194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:03:18.599680 20194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:03:18.607726 20194 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:43057
I20260806 17:03:18.608021 20265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:43057 every 8 connection(s)
I20260806 17:03:18.608749 20194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:03:18.610075 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 20194
I20260806 17:03:18.610201 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:03:18.610481 9661 external_mini_cluster.cc:1428] Setting key cb4219cf870a4a0188bfcb9464f473a1
I20260806 17:03:18.612128 20266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:03:18.619120 20266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31: Bootstrap starting.
Aug 06 17:03:18 dist-test-slave-69rg krb5kdc[20180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:03:18.621496 20266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31: Neither blocks nor log segments found. Creating new log.
I20260806 17:03:18.622303 20266 log.cc:824] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31: Log is configured to *not* fsync() on all Append() calls
I20260806 17:03:18.624276 20266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31: No bootstrap required, opened a new log
I20260806 17:03:18.625610 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:18.611975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46622 (local address 127.9.111.126:43057)
0806 17:03:18.612395 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:18.612411 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:18.612445 (+ 34us) server_negotiation.cc:407] Connection header received
0806 17:03:18.613104 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:18.613120 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:18.613442 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:18.613723 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:18.614774 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:18.615892 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:18.617059 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:18.617546 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:18.617675 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:18.620704 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:03:18.620709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:03:18.620723 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:03:18.620752 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:03:18.622637 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:03:18.623102 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:03:18.623107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:03:18.623111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:03:18.623225 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:03:18.623512 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:03:18.623516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:03:18.623518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:03:18.623917 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:03:18.624127 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:18.624412 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 17:03:18.624722 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":138,"threads_started":1}
I20260806 17:03:18.628073 20266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac067dab29a44f5b6945b68ef92fb31" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43057 } }
I20260806 17:03:18.628455 20266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:03:18.628537 20266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac067dab29a44f5b6945b68ef92fb31, State: Initialized, Role: FOLLOWER
I20260806 17:03:18.629083 20266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac067dab29a44f5b6945b68ef92fb31" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43057 } }
I20260806 17:03:18.629256 20266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:03:18.629339 20266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:03:18.629444 20266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:03:18.630825 20266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac067dab29a44f5b6945b68ef92fb31" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43057 } }
I20260806 17:03:18.631213 20266 leader_election.cc:302] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ac067dab29a44f5b6945b68ef92fb31; no voters:
I20260806 17:03:18.631562 20266 leader_election.cc:288] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:03:18.631693 20271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:03:18.631965 20271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31 [term 1 LEADER]: Becoming Leader. State: Replica: 2ac067dab29a44f5b6945b68ef92fb31, State: Running, Role: LEADER
I20260806 17:03:18.632447 20271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac067dab29a44f5b6945b68ef92fb31" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43057 } }
I20260806 17:03:18.633039 20266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 17:03:18.639140 20285 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:03:18.639329 20285 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:03:18.635866 20272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ac067dab29a44f5b6945b68ef92fb31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac067dab29a44f5b6945b68ef92fb31" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43057 } } }
I20260806 17:03:18.640867 20272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31 [sys.catalog]: This master's current role is: LEADER
I20260806 17:03:18.636299 20273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ac067dab29a44f5b6945b68ef92fb31. Latest consensus state: current_term: 1 leader_uuid: "2ac067dab29a44f5b6945b68ef92fb31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac067dab29a44f5b6945b68ef92fb31" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43057 } } }
I20260806 17:03:18.641131 20273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31 [sys.catalog]: This master's current role is: LEADER
I20260806 17:03:18.641342 20288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:03:18.643049 20288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:03:18.649307 20288 catalog_manager.cc:1347] Generated new cluster ID: b8269f0fb8404081a5d63854b8a4a183
I20260806 17:03:18.649395 20288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:03:18.670540 20288 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:03:18.671790 20288 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:03:18.685967 20288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31: Generated new TSK 0
I20260806 17:03:18.686770 20288 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
I20260806 17:03:18.729607 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:43057
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:03:18.845836 20292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:03:18.846189 20292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:03:18.846266 20292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:03:18.850447 20292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:03:18.850517 20292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:03:18.850579 20292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:03:18.856207 20292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:43057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:03:18.857350 20292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:03:18.858421 20292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:03:18.868585 20297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:03:18.868654 20292 server_base.cc:1034] running on GCE node
W20260806 17:03:18.868595 20300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:18.868615 20298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:03:18.869288 20292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:03:18.870064 20292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:03:18.871243 20292 hybrid_clock.cc:648] HybridClock initialized: now 1786035798871204 us; error 48 us; skew 500 ppm
Aug 06 17:03:18 dist-test-slave-69rg krb5kdc[20180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035798, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:03:18.874682 20292 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:03:18.875717 20292 webserver.cc:460] Webserver started at http://127.9.111.65:39267/ using document root <none> and password file <none>
I20260806 17:03:18.876309 20292 fs_manager.cc:359] Metadata directory not provided
I20260806 17:03:18.876374 20292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:03:18.876595 20292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:03:18.878818 20292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "46b89663ae6a40b09d53594f88dbd0c5"
format_stamp: "Formatted at 2026-08-06 17:03:18 on dist-test-slave-69rg"
server_key: "89c9c53b967bd655da56c08d4dbe4639"
server_key_iv: "a3e1b0997ab9c95f7f8212a541e9c3ed"
server_key_version: "encryptionkey@0"
I20260806 17:03:18.879381 20292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "46b89663ae6a40b09d53594f88dbd0c5"
format_stamp: "Formatted at 2026-08-06 17:03:18 on dist-test-slave-69rg"
server_key: "89c9c53b967bd655da56c08d4dbe4639"
server_key_iv: "a3e1b0997ab9c95f7f8212a541e9c3ed"
server_key_version: "encryptionkey@0"
I20260806 17:03:18.883252 20292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:03:18.886032 20307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:18.887730 20292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:03:18.887903 20292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "46b89663ae6a40b09d53594f88dbd0c5"
format_stamp: "Formatted at 2026-08-06 17:03:18 on dist-test-slave-69rg"
server_key: "89c9c53b967bd655da56c08d4dbe4639"
server_key_iv: "a3e1b0997ab9c95f7f8212a541e9c3ed"
server_key_version: "encryptionkey@0"
I20260806 17:03:18.888048 20292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:03:18.911067 20292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:03:18.915009 20292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:03:18.915293 20292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:03:18.915957 20292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:03:18.916857 20292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:03:18.916936 20292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:18.917012 20292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:03:18.917057 20292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:18.929407 20292 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:33233
I20260806 17:03:18.929716 20425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:33233 every 8 connection(s)
I20260806 17:03:18.930505 20292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:03:18.937996 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 20292
I20260806 17:03:18.938151 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:03:18.938459 9661 external_mini_cluster.cc:1428] Setting key a3e3ef11bc51fc7ff07ceaa767946c13
Aug 06 17:03:18 dist-test-slave-69rg krb5kdc[20180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035798, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:03:18.948195 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:18.932745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:59545 (local address 127.9.111.126:43057)
0806 17:03:18.932891 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:18.932895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:18.933945 (+ 1050us) server_negotiation.cc:407] Connection header received
0806 17:03:18.935890 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:18.935894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:18.935965 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:18.936048 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:18.937386 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:18.938177 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:18.939826 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:18.940080 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:18.940206 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:18.943318 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:03:18.943325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:03:18.943331 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:03:18.943373 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:03:18.945333 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:03:18.946076 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:03:18.946082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:03:18.946085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:03:18.946201 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:03:18.946639 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:03:18.946643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:03:18.946645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:03:18.946956 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:03:18.947080 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:18.947761 (+ 681us) server_negotiation.cc:299] Negotiation successful
0806 17:03:18.947938 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 17:03:18.948918 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:18.933133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:43057 (local address 127.9.111.65:59545)
0806 17:03:18.933694 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:18.933742 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:18.935613 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:18.936253 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:18.936261 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:18.936672 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:18.937192 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:18.937206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:18.938368 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:18.938372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:18.939656 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:18.939668 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:18.940268 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:18.940273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:18.940456 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:18.941158 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:18.941176 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:18.943078 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:03:18.945520 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:03:18.945528 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:03:18.945550 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:03:18.945954 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:03:18.946334 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:03:18.946337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:03:18.946339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:03:18.946508 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:03:18.947103 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:03:18.947109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:03:18.947437 (+ 328us) client_negotiation.cc:769] Sending connection context
0806 17:03:18.947619 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 17:03:18.947928 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":129,"threads_started":1}
I20260806 17:03:18.950502 20426 heartbeater.cc:347] Connected to a master server at 127.9.111.126:43057
I20260806 17:03:18.950796 20426 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.111.66@KRBTEST.COM; defaulting to no policy
I20260806 17:03:18.951396 20426 heartbeater.cc:511] Master 127.9.111.126:43057 requested a full tablet report, sending...
I20260806 17:03:18.953423 20229 ts_manager.cc:194] Registered new tserver with Master: 46b89663ae6a40b09d53594f88dbd0c5 (127.9.111.65:33233)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.66@KRBTEST.COM" created.
I20260806 17:03:18.954957 20229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:59545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
I20260806 17:03:18.980232 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:43057
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:03:19.112746 20431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:03:19.113090 20431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:03:19.113140 20431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:03:19.117733 20431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:03:19.117832 20431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:03:19.117900 20431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:03:19.123836 20431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:43057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:03:19.124946 20431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:03:19.126142 20431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:03:19.137943 20439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:19.137909 20436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:19.138006 20437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:03:19.138151 20431 server_base.cc:1034] running on GCE node
I20260806 17:03:19.138772 20431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:03:19.139525 20431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:03:19.140736 20431 hybrid_clock.cc:648] HybridClock initialized: now 1786035799140717 us; error 34 us; skew 500 ppm
Aug 06 17:03:19 dist-test-slave-69rg krb5kdc[20180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035799, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:03:19.143939 20431 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:03:19.144943 20431 webserver.cc:460] Webserver started at http://127.9.111.66:39669/ using document root <none> and password file <none>
I20260806 17:03:19.145556 20431 fs_manager.cc:359] Metadata directory not provided
I20260806 17:03:19.145656 20431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:03:19.145968 20431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:03:19.148270 20431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "9797a583827f40c5a6166a11df05a2d0"
format_stamp: "Formatted at 2026-08-06 17:03:19 on dist-test-slave-69rg"
server_key: "f503a27391e15263a90a62db9cefa1ec"
server_key_iv: "d93d99bdb574f8a46d77bb85bb4abcdb"
server_key_version: "encryptionkey@0"
I20260806 17:03:19.148907 20431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "9797a583827f40c5a6166a11df05a2d0"
format_stamp: "Formatted at 2026-08-06 17:03:19 on dist-test-slave-69rg"
server_key: "f503a27391e15263a90a62db9cefa1ec"
server_key_iv: "d93d99bdb574f8a46d77bb85bb4abcdb"
server_key_version: "encryptionkey@0"
I20260806 17:03:19.153157 20431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 17:03:19.156087 20446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:19.157573 20431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 17:03:19.157742 20431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "9797a583827f40c5a6166a11df05a2d0"
format_stamp: "Formatted at 2026-08-06 17:03:19 on dist-test-slave-69rg"
server_key: "f503a27391e15263a90a62db9cefa1ec"
server_key_iv: "d93d99bdb574f8a46d77bb85bb4abcdb"
server_key_version: "encryptionkey@0"
I20260806 17:03:19.157951 20431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:03:19.175884 20431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:03:19.179991 20431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:03:19.180243 20431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:03:19.180891 20431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:03:19.181913 20431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:03:19.181983 20431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:19.182060 20431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:03:19.182103 20431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:19.193466 20431 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:42737
I20260806 17:03:19.193485 20562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:42737 every 8 connection(s)
I20260806 17:03:19.194562 20431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:03:19.199216 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 20431
I20260806 17:03:19.199380 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:03:19.199657 9661 external_mini_cluster.cc:1428] Setting key df298859bbcb7849832048f1b6c58bc6
Aug 06 17:03:19 dist-test-slave-69rg krb5kdc[20180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035799, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.111.67@KRBTEST.COM; defaulting to no policy
I20260806 17:03:19.212357 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:19.196559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:56283 (local address 127.9.111.126:43057)
0806 17:03:19.196739 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:19.196743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:19.197861 (+ 1118us) server_negotiation.cc:407] Connection header received
0806 17:03:19.199351 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:19.199355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:19.199427 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:19.199515 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:19.201108 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.201775 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.203282 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.203711 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.203854 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:19.207608 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:03:19.207612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:03:19.207615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:03:19.207648 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:03:19.209487 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:03:19.210326 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:03:19.210330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:03:19.210331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:03:19.210425 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:03:19.210952 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:03:19.210955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:03:19.210956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:03:19.211214 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:03:19.211315 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:19.212097 (+ 782us) server_negotiation.cc:299] Negotiation successful
0806 17:03:19.212227 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 17:03:19.213304 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:19.197046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:43057 (local address 127.9.111.66:56283)
0806 17:03:19.197623 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:19.197674 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:19.199090 (+ 1416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:19.199725 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:19.199759 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:19.200182 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:19.200919 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:19.200936 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.201996 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.202000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:19.203022 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:19.203034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.204253 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.204262 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:19.204457 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:19.205187 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:19.205206 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:19.207292 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:03:19.209657 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:03:19.209667 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:03:19.209691 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:03:19.210178 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:03:19.210547 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:03:19.210553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:03:19.210555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:03:19.210789 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:03:19.211348 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:03:19.211357 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:03:19.211772 (+ 415us) client_negotiation.cc:769] Sending connection context
0806 17:03:19.211966 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 17:03:19.212272 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":151,"threads_started":1}
I20260806 17:03:19.215039 20563 heartbeater.cc:347] Connected to a master server at 127.9.111.126:43057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.67@KRBTEST.COM" created.
I20260806 17:03:19.215397 20563 heartbeater.cc:464] Registering TS with master...
I20260806 17:03:19.216566 20563 heartbeater.cc:511] Master 127.9.111.126:43057 requested a full tablet report, sending...
I20260806 17:03:19.218075 20229 ts_manager.cc:194] Registered new tserver with Master: 9797a583827f40c5a6166a11df05a2d0 (127.9.111.66:42737)
I20260806 17:03:19.219036 20229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:56283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
I20260806 17:03:19.239519 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:43057
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:03:19.363111 20568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:03:19.363459 20568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:03:19.363511 20568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:03:19.368467 20568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:03:19.368551 20568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:03:19.368664 20568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:03:19.375563 20568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:43057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:03:19.376845 20568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:03:19.378011 20568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:03:19.389261 20575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:19.389315 20578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:03:19.389372 20574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:03:19.389484 20568 server_base.cc:1034] running on GCE node
I20260806 17:03:19.390275 20568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:03:19.391178 20568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:03:19.392458 20568 hybrid_clock.cc:648] HybridClock initialized: now 1786035799392433 us; error 41 us; skew 500 ppm
Aug 06 17:03:19 dist-test-slave-69rg krb5kdc[20180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035799, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:03:19.395975 20568 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:03:19.397078 20568 webserver.cc:460] Webserver started at http://127.9.111.67:38021/ using document root <none> and password file <none>
I20260806 17:03:19.397696 20568 fs_manager.cc:359] Metadata directory not provided
I20260806 17:03:19.397804 20568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:03:19.398154 20568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:03:19.400344 20568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "7c24feac5d084c9aa5cd8f931399a478"
format_stamp: "Formatted at 2026-08-06 17:03:19 on dist-test-slave-69rg"
server_key: "5f587dbdd0e5b9586526f274f84c07d2"
server_key_iv: "bd1a5785f36b8edce7120fda60203e4e"
server_key_version: "encryptionkey@0"
I20260806 17:03:19.400893 20568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "7c24feac5d084c9aa5cd8f931399a478"
format_stamp: "Formatted at 2026-08-06 17:03:19 on dist-test-slave-69rg"
server_key: "5f587dbdd0e5b9586526f274f84c07d2"
server_key_iv: "bd1a5785f36b8edce7120fda60203e4e"
server_key_version: "encryptionkey@0"
I20260806 17:03:19.404762 20568 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260806 17:03:19.407378 20585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:19.408771 20568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 17:03:19.408929 20568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "7c24feac5d084c9aa5cd8f931399a478"
format_stamp: "Formatted at 2026-08-06 17:03:19 on dist-test-slave-69rg"
server_key: "5f587dbdd0e5b9586526f274f84c07d2"
server_key_iv: "bd1a5785f36b8edce7120fda60203e4e"
server_key_version: "encryptionkey@0"
I20260806 17:03:19.409051 20568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:03:19.438802 20568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:03:19.442346 20568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:03:19.442566 20568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:03:19.443216 20568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:03:19.444144 20568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:03:19.444219 20568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:19.444294 20568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:03:19.444339 20568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:03:19.454341 20568 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:36135
I20260806 17:03:19.454376 20706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:36135 every 8 connection(s)
I20260806 17:03:19.455325 20568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:03:19.458963 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 20568
I20260806 17:03:19.459122 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:03:19.459489 9661 external_mini_cluster.cc:1428] Setting key 75725797facf93724f0cd85ed2662df8
Aug 06 17:03:19 dist-test-slave-69rg krb5kdc[20180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035799, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:03:19.471141 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:19.457487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:43005 (local address 127.9.111.126:43057)
0806 17:03:19.457656 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:19.457662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:19.458590 (+ 928us) server_negotiation.cc:407] Connection header received
0806 17:03:19.460322 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:19.460324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:19.460402 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:19.460486 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:19.462163 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.462784 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.463959 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.464177 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.464289 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:19.466936 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:03:19.466939 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:03:19.466941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:03:19.466968 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:03:19.468573 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:03:19.469207 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:03:19.469210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:03:19.469212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:03:19.469302 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:03:19.469794 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:03:19.469797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:03:19.469798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:03:19.470100 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:03:19.470189 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:19.470797 (+ 608us) server_negotiation.cc:299] Negotiation successful
0806 17:03:19.470997 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 17:03:19.472224 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:19.457958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:43057 (local address 127.9.111.67:43005)
0806 17:03:19.458430 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:19.458475 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:19.460082 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:19.460683 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:19.460690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:19.461279 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:03:19.462003 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:19.462020 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.462910 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.462913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:19.463836 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:19.463846 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.464359 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.464362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:19.464521 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:19.465103 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:03:19.465120 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:03:19.466740 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:03:19.468732 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:03:19.468739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:03:19.468753 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:03:19.469043 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:03:19.469424 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:03:19.469427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:03:19.469428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:03:19.469621 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:03:19.470253 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:03:19.470259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:03:19.470542 (+ 283us) client_negotiation.cc:769] Sending connection context
0806 17:03:19.470727 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 17:03:19.471173 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":127,"threads_started":1}
I20260806 17:03:19.473965 20707 heartbeater.cc:347] Connected to a master server at 127.9.111.126:43057
I20260806 17:03:19.474238 20707 heartbeater.cc:464] Registering TS with master...
I20260806 17:03:19.474843 20707 heartbeater.cc:511] Master 127.9.111.126:43057 requested a full tablet report, sending...
I20260806 17:03:19.475937 20229 ts_manager.cc:194] Registered new tserver with Master: 7c24feac5d084c9aa5cd8f931399a478 (127.9.111.67:36135)
I20260806 17:03:19.476547 20229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:43005
I20260806 17:03:19.487808 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 17:03:19.499257 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:19.490239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.9.111.126:43057)
0806 17:03:19.490434 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:19.490439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:19.490552 (+ 113us) server_negotiation.cc:407] Connection header received
0806 17:03:19.490631 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:19.490633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:19.490682 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:19.490748 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:03:19.491618 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.492182 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.493287 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.493560 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.493665 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:19.495462 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:03:19.495466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:03:19.495468 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:03:19.495500 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:03:19.497115 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:03:19.497629 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:03:19.497633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:03:19.497635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:03:19.497740 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:03:19.498196 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:03:19.498199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:03:19.498200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:03:19.498458 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:03:19.498576 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:19.498846 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 17:03:19.499018 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 17:03:19.502847 20229 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:03:19.506176 20229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:03:19.524417 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:19.517978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:36135 (local address 127.0.0.1:51488)
0806 17:03:19.518783 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:19.518798 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:19.518898 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:19.519544 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:19.519547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:19.519577 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:03:19.519679 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:19.519689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.521369 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.521372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:19.522679 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:19.522685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.523716 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.523720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:19.523894 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:19.523959 (+ 65us) client_negotiation.cc:769] Sending connection context
0806 17:03:19.524079 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 17:03:19.524229 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":103,"spinlock_wait_cycles":10112,"thread_start_us":137,"threads_started":1}
I20260806 17:03:19.525066 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:19.517259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:42737 (local address 127.0.0.1:37412)
0806 17:03:19.518364 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:19.518380 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:19.518475 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:19.519512 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:19.519517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:19.519539 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:03:19.519663 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:19.519671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.521510 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.521516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:19.522875 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:19.522886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.524111 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.524115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:19.524220 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:19.524241 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 17:03:19.524292 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 17:03:19.524398 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"mutex_wait_us":220,"thread_start_us":130,"threads_started":1}
I20260806 17:03:19.525087 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:19.518437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51488 (local address 127.9.111.67:36135)
0806 17:03:19.519055 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:19.519065 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:19.519083 (+ 18us) server_negotiation.cc:407] Connection header received
0806 17:03:19.519208 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:19.519213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:19.519392 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:19.519544 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:03:19.519820 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.521213 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.522855 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.523556 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.523655 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:19.523797 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:19.524806 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0806 17:03:19.524950 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":259,"threads_started":1}
I20260806 17:03:19.526021 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:19.518213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.9.111.66:42737)
0806 17:03:19.518919 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:19.518929 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:19.518952 (+ 23us) server_negotiation.cc:407] Connection header received
0806 17:03:19.519022 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:19.519026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:19.519188 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:19.519848 (+ 660us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:03:19.520107 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.521219 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.523073 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.523953 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.524508 (+ 555us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:19.525650 (+ 1142us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:19.525716 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 17:03:19.525858 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":110,"threads_started":1}
I20260806 17:03:19.526140 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:19.517143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:33233 (local address 127.0.0.1:41648)
0806 17:03:19.517810 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:19.517901 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:19.518049 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:19.519219 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:19.519226 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:19.519251 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:03:19.519414 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:19.519422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.521006 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.521011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:19.522668 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:19.522683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.525617 (+ 2934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.525621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:19.525699 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:19.525718 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 17:03:19.525903 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 17:03:19.525993 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":30,"thread_start_us":100,"threads_started":1}
I20260806 17:03:19.526932 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:19.517262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41648 (local address 127.9.111.65:33233)
0806 17:03:19.518536 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:19.518544 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:19.518573 (+ 29us) server_negotiation.cc:407] Connection header received
0806 17:03:19.518791 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:19.518798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:19.519044 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:19.519353 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:03:19.519549 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.520874 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.523884 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.524997 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.525292 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:19.525476 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:19.525958 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 17:03:19.526072 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":120,"threads_started":1}
I20260806 17:03:19.528851 20638 tablet_service.cc:1467] Processing CreateTablet for tablet b1ddf0678a1e480d834e0a374fa8caf3 (DEFAULT_TABLE table=test-table [id=f38cf31f8a33404fbc65cbdb2d89b221]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:03:19.529495 20495 tablet_service.cc:1467] Processing CreateTablet for tablet b1ddf0678a1e480d834e0a374fa8caf3 (DEFAULT_TABLE table=test-table [id=f38cf31f8a33404fbc65cbdb2d89b221]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:03:19.530263 20638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1ddf0678a1e480d834e0a374fa8caf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:03:19.530658 20495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1ddf0678a1e480d834e0a374fa8caf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:03:19.530294 20354 tablet_service.cc:1467] Processing CreateTablet for tablet b1ddf0678a1e480d834e0a374fa8caf3 (DEFAULT_TABLE table=test-table [id=f38cf31f8a33404fbc65cbdb2d89b221]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:03:19.531365 20354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1ddf0678a1e480d834e0a374fa8caf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:03:19.537969 20725 tablet_bootstrap.cc:492] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478: Bootstrap starting.
I20260806 17:03:19.539903 20726 tablet_bootstrap.cc:492] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5: Bootstrap starting.
I20260806 17:03:19.542152 20727 tablet_bootstrap.cc:492] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0: Bootstrap starting.
I20260806 17:03:19.543124 20725 tablet_bootstrap.cc:654] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478: Neither blocks nor log segments found. Creating new log.
I20260806 17:03:19.543354 20726 tablet_bootstrap.cc:654] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5: Neither blocks nor log segments found. Creating new log.
I20260806 17:03:19.544298 20725 log.cc:824] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478: Log is configured to *not* fsync() on all Append() calls
I20260806 17:03:19.544287 20726 log.cc:824] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5: Log is configured to *not* fsync() on all Append() calls
I20260806 17:03:19.545781 20727 tablet_bootstrap.cc:654] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0: Neither blocks nor log segments found. Creating new log.
I20260806 17:03:19.546784 20727 log.cc:824] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0: Log is configured to *not* fsync() on all Append() calls
I20260806 17:03:19.546916 20725 tablet_bootstrap.cc:492] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478: No bootstrap required, opened a new log
I20260806 17:03:19.546916 20726 tablet_bootstrap.cc:492] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5: No bootstrap required, opened a new log
I20260806 17:03:19.547154 20726 ts_tablet_manager.cc:1397] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 17:03:19.547154 20725 ts_tablet_manager.cc:1397] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260806 17:03:19.549126 20727 tablet_bootstrap.cc:492] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0: No bootstrap required, opened a new log
I20260806 17:03:19.549376 20727 ts_tablet_manager.cc:1397] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 17:03:19.551887 20725 raft_consensus.cc:348] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b89663ae6a40b09d53594f88dbd0c5" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33233 } } peers { permanent_uuid: "7c24feac5d084c9aa5cd8f931399a478" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36135 } } peers { permanent_uuid: "9797a583827f40c5a6166a11df05a2d0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 42737 } }
I20260806 17:03:19.552271 20725 raft_consensus.cc:374] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:03:19.552361 20725 raft_consensus.cc:729] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c24feac5d084c9aa5cd8f931399a478, State: Initialized, Role: FOLLOWER
I20260806 17:03:19.552285 20726 raft_consensus.cc:348] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b89663ae6a40b09d53594f88dbd0c5" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33233 } } peers { permanent_uuid: "7c24feac5d084c9aa5cd8f931399a478" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36135 } } peers { permanent_uuid: "9797a583827f40c5a6166a11df05a2d0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 42737 } }
I20260806 17:03:19.552567 20726 raft_consensus.cc:374] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:03:19.552623 20726 raft_consensus.cc:729] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46b89663ae6a40b09d53594f88dbd0c5, State: Initialized, Role: FOLLOWER
I20260806 17:03:19.552505 20727 raft_consensus.cc:348] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b89663ae6a40b09d53594f88dbd0c5" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33233 } } peers { permanent_uuid: "7c24feac5d084c9aa5cd8f931399a478" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36135 } } peers { permanent_uuid: "9797a583827f40c5a6166a11df05a2d0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 42737 } }
I20260806 17:03:19.552953 20725 consensus_queue.cc:260] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b89663ae6a40b09d53594f88dbd0c5" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33233 } } peers { permanent_uuid: "7c24feac5d084c9aa5cd8f931399a478" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36135 } } peers { permanent_uuid: "9797a583827f40c5a6166a11df05a2d0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 42737 } }
I20260806 17:03:19.553000 20726 consensus_queue.cc:260] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b89663ae6a40b09d53594f88dbd0c5" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33233 } } peers { permanent_uuid: "7c24feac5d084c9aa5cd8f931399a478" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36135 } } peers { permanent_uuid: "9797a583827f40c5a6166a11df05a2d0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 42737 } }
I20260806 17:03:19.553088 20727 raft_consensus.cc:374] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:03:19.553154 20727 raft_consensus.cc:729] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9797a583827f40c5a6166a11df05a2d0, State: Initialized, Role: FOLLOWER
I20260806 17:03:19.553656 20727 consensus_queue.cc:260] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b89663ae6a40b09d53594f88dbd0c5" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33233 } } peers { permanent_uuid: "7c24feac5d084c9aa5cd8f931399a478" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36135 } } peers { permanent_uuid: "9797a583827f40c5a6166a11df05a2d0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 42737 } }
I20260806 17:03:19.554255 20426 heartbeater.cc:503] Master 127.9.111.126:43057 was elected leader, sending a full tablet report...
I20260806 17:03:19.554328 20707 heartbeater.cc:503] Master 127.9.111.126:43057 was elected leader, sending a full tablet report...
I20260806 17:03:19.554414 20725 ts_tablet_manager.cc:1428] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478: Time spent starting tablet: real 0.007s user 0.008s sys 0.001s
I20260806 17:03:19.555048 20727 ts_tablet_manager.cc:1428] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 17:03:19.555120 20563 heartbeater.cc:503] Master 127.9.111.126:43057 was elected leader, sending a full tablet report...
I20260806 17:03:19.555557 20726 ts_tablet_manager.cc:1428] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20260806 17:03:19.682188 20427 tablet.cc:2369] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 17:03:19.695878 20564 tablet.cc:2369] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 17:03:19.706455 20708 tablet.cc:2369] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:03:19.873967 20732 raft_consensus.cc:482] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:03:19.874244 20732 raft_consensus.cc:504] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b89663ae6a40b09d53594f88dbd0c5" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33233 } } peers { permanent_uuid: "7c24feac5d084c9aa5cd8f931399a478" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36135 } } peers { permanent_uuid: "9797a583827f40c5a6166a11df05a2d0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 42737 } }
I20260806 17:03:19.875530 20732 leader_election.cc:288] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c24feac5d084c9aa5cd8f931399a478 (127.9.111.67:36135), 9797a583827f40c5a6166a11df05a2d0 (127.9.111.66:42737)
I20260806 17:03:19.881560 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:19.875984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:36689 (local address 127.9.111.66:42737)
0806 17:03:19.876120 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:19.876125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:19.876439 (+ 314us) server_negotiation.cc:407] Connection header received
0806 17:03:19.876497 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:19.876499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:19.876553 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:19.876646 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:03:19.877231 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.878027 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.880089 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.880717 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.880854 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:19.880909 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:19.881329 (+ 420us) server_negotiation.cc:299] Negotiation successful
0806 17:03:19.881415 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 17:03:19.881560 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:19.875920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:48767 (local address 127.9.111.67:36135)
0806 17:03:19.876079 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:19.876086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:19.876401 (+ 315us) server_negotiation.cc:407] Connection header received
0806 17:03:19.876487 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:19.876490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:19.876549 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:19.876645 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:03:19.877183 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.878041 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.880089 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.880712 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.880854 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:19.880908 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:19.881329 (+ 421us) server_negotiation.cc:299] Negotiation successful
0806 17:03:19.881415 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 17:03:19.881650 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:19.875907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:42737 (local address 127.9.111.65:36689)
0806 17:03:19.876276 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:19.876290 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:19.876385 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:19.876719 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:19.876723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:19.876779 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:03:19.877015 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:19.877023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.878193 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.878197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:19.879917 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:19.879925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.880857 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.880861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:19.881144 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:19.881182 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 17:03:19.881272 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 17:03:19.881410 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":367,"spinlock_wait_cycles":31616,"thread_start_us":121,"threads_started":1}
I20260806 17:03:19.881650 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:19.875781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:36135 (local address 127.9.111.65:48767)
0806 17:03:19.876205 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:19.876221 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:03:19.876375 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:03:19.876719 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:03:19.876723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:03:19.876770 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:03:19.877015 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:19.877023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.878193 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.878197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:19.879910 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:03:19.879925 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.880857 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.880861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:03:19.881115 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:19.881169 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 17:03:19.881272 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 17:03:19.881410 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":358,"spinlock_wait_cycles":11136,"thread_start_us":142,"threads_started":1}
I20260806 17:03:19.882414 20659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ddf0678a1e480d834e0a374fa8caf3" candidate_uuid: "46b89663ae6a40b09d53594f88dbd0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c24feac5d084c9aa5cd8f931399a478" is_pre_election: true
I20260806 17:03:19.882431 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ddf0678a1e480d834e0a374fa8caf3" candidate_uuid: "46b89663ae6a40b09d53594f88dbd0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9797a583827f40c5a6166a11df05a2d0" is_pre_election: true
I20260806 17:03:19.882751 20659 raft_consensus.cc:2454] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46b89663ae6a40b09d53594f88dbd0c5 in term 0.
I20260806 17:03:19.882767 20516 raft_consensus.cc:2454] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46b89663ae6a40b09d53594f88dbd0c5 in term 0.
I20260806 17:03:19.883289 20310 leader_election.cc:302] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 46b89663ae6a40b09d53594f88dbd0c5, 7c24feac5d084c9aa5cd8f931399a478, 9797a583827f40c5a6166a11df05a2d0; no voters:
I20260806 17:03:19.883621 20732 raft_consensus.cc:2781] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:03:19.883708 20732 raft_consensus.cc:482] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:03:19.883750 20732 raft_consensus.cc:3037] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:03:19.884859 20732 raft_consensus.cc:504] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b89663ae6a40b09d53594f88dbd0c5" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33233 } } peers { permanent_uuid: "7c24feac5d084c9aa5cd8f931399a478" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36135 } } peers { permanent_uuid: "9797a583827f40c5a6166a11df05a2d0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 42737 } }
I20260806 17:03:19.885251 20732 leader_election.cc:288] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [CANDIDATE]: Term 1 election: Requested vote from peers 7c24feac5d084c9aa5cd8f931399a478 (127.9.111.67:36135), 9797a583827f40c5a6166a11df05a2d0 (127.9.111.66:42737)
I20260806 17:03:19.885684 20659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ddf0678a1e480d834e0a374fa8caf3" candidate_uuid: "46b89663ae6a40b09d53594f88dbd0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c24feac5d084c9aa5cd8f931399a478"
I20260806 17:03:19.885756 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ddf0678a1e480d834e0a374fa8caf3" candidate_uuid: "46b89663ae6a40b09d53594f88dbd0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9797a583827f40c5a6166a11df05a2d0"
I20260806 17:03:19.885862 20659 raft_consensus.cc:3037] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:03:19.885946 20516 raft_consensus.cc:3037] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:03:19.886945 20659 raft_consensus.cc:2454] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46b89663ae6a40b09d53594f88dbd0c5 in term 1.
I20260806 17:03:19.887036 20516 raft_consensus.cc:2454] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46b89663ae6a40b09d53594f88dbd0c5 in term 1.
I20260806 17:03:19.887262 20310 leader_election.cc:302] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46b89663ae6a40b09d53594f88dbd0c5, 7c24feac5d084c9aa5cd8f931399a478; no voters:
I20260806 17:03:19.887485 20732 raft_consensus.cc:2781] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:03:19.887764 20732 raft_consensus.cc:686] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [term 1 LEADER]: Becoming Leader. State: Replica: 46b89663ae6a40b09d53594f88dbd0c5, State: Running, Role: LEADER
I20260806 17:03:19.888149 20732 consensus_queue.cc:237] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b89663ae6a40b09d53594f88dbd0c5" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33233 } } peers { permanent_uuid: "7c24feac5d084c9aa5cd8f931399a478" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36135 } } peers { permanent_uuid: "9797a583827f40c5a6166a11df05a2d0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 42737 } }
I20260806 17:03:19.891543 20224 catalog_manager.cc:5495] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46b89663ae6a40b09d53594f88dbd0c5 (127.9.111.65). New cstate: current_term: 1 leader_uuid: "46b89663ae6a40b09d53594f88dbd0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b89663ae6a40b09d53594f88dbd0c5" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c24feac5d084c9aa5cd8f931399a478" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9797a583827f40c5a6166a11df05a2d0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 42737 } health_report { overall_health: UNKNOWN } } }
I20260806 17:03:19.981599 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:19.977746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.9.111.65:33233)
0806 17:03:19.977992 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 17:03:19.977996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:03:19.978013 (+ 17us) server_negotiation.cc:407] Connection header received
0806 17:03:19.978111 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:03:19.978113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:03:19.978216 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:03:19.978319 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:03:19.978588 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.979129 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.979991 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:03:19.980210 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:03:19.980307 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:03:19.980679 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:03:19.981209 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:03:19.981315 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 17:03:19.981426 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 17:03:19.981481 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 17:03:19.987804 20659 raft_consensus.cc:1261] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478 [term 1 FOLLOWER]: Refusing update from remote peer 46b89663ae6a40b09d53594f88dbd0c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:03:19.987848 20516 raft_consensus.cc:1261] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0 [term 1 FOLLOWER]: Refusing update from remote peer 46b89663ae6a40b09d53594f88dbd0c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:03:19.988579 20732 consensus_queue.cc:1035] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c24feac5d084c9aa5cd8f931399a478" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 36135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:03:19.988801 20736 consensus_queue.cc:1035] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9797a583827f40c5a6166a11df05a2d0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 42737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:03:19.997232 20739 mvcc.cc:205] Tried to move back new op lower bound from 7315602636742324224 to 7315602636344393728. Current Snapshot: MvccSnapshot[applied={T|T < 7315602636742324224}]
I20260806 17:03:19.997885 20740 mvcc.cc:205] Tried to move back new op lower bound from 7315602636742324224 to 7315602636344393728. Current Snapshot: MvccSnapshot[applied={T|T < 7315602636742324224}]
I20260806 17:03:20.001608 20741 mvcc.cc:205] Tried to move back new op lower bound from 7315602636742324224 to 7315602636344393728. Current Snapshot: MvccSnapshot[applied={T|T < 7315602636742324224 or (T in {7315602636742324224})}]
I20260806 17:03:20.007125 20225 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:03:20.007371 20225 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:03:20.010560 20225 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ac067dab29a44f5b6945b68ef92fb31: Sending DeleteTablet for 3 replicas of tablet b1ddf0678a1e480d834e0a374fa8caf3
I20260806 17:03:20.011749 20495 tablet_service.cc:1514] Processing DeleteTablet for tablet b1ddf0678a1e480d834e0a374fa8caf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:03:20 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:37412
I20260806 17:03:20.011770 20638 tablet_service.cc:1514] Processing DeleteTablet for tablet b1ddf0678a1e480d834e0a374fa8caf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:03:20 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:51488
I20260806 17:03:20.012118 20354 tablet_service.cc:1514] Processing DeleteTablet for tablet b1ddf0678a1e480d834e0a374fa8caf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:03:20 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:41648
I20260806 17:03:20.012460 20750 tablet_replica.cc:330] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0: stopping tablet replica
I20260806 17:03:20.012813 20751 tablet_replica.cc:330] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5: stopping tablet replica
I20260806 17:03:20.012991 20750 raft_consensus.cc:2229] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:03:20.013273 20751 raft_consensus.cc:2229] T b1ddf0678a1e480d834e0a374fa8caf3 P 46b89663ae6a40b09d53594f88dbd0c5 [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:03:20.013303 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 20292
I20260806 17:03:20.013427 20750 raft_consensus.cc:2258] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:03:20.014314 20749 tablet_replica.cc:330] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478: stopping tablet replica
I20260806 17:03:20.014796 20749 raft_consensus.cc:2229] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:03:20.016376 20750 ts_tablet_manager.cc:1905] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:03:20.015164 20749 raft_consensus.cc:2258] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:03:20.018970 20750 ts_tablet_manager.cc:1918] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:03:20.019089 20750 log.cc:1196] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/b1ddf0678a1e480d834e0a374fa8caf3
I20260806 17:03:20.019487 20750 ts_tablet_manager.cc:1939] T b1ddf0678a1e480d834e0a374fa8caf3 P 9797a583827f40c5a6166a11df05a2d0: Deleting consensus metadata
I20260806 17:03:20.020637 20749 ts_tablet_manager.cc:1905] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:03:20.021145 20211 catalog_manager.cc:4841] TS 9797a583827f40c5a6166a11df05a2d0 (127.9.111.66:42737): tablet b1ddf0678a1e480d834e0a374fa8caf3 (table test-table [id=f38cf31f8a33404fbc65cbdb2d89b221]) successfully deleted
W20260806 17:03:20.021983 20212 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.111.65:33233 (error 108)
I20260806 17:03:20.022142 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 20431
I20260806 17:03:20.023948 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:03:20.023638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:33233 (local address 127.0.0.1:41660)
0806 17:03:20.023803 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 17:03:20.023871 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.65:33233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 17:03:20.024178 20212 catalog_manager.cc:4584] TS 46b89663ae6a40b09d53594f88dbd0c5 (127.9.111.65:33233): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b1ddf0678a1e480d834e0a374fa8caf3: Network error: Client connection negotiation failed: client connection to 127.9.111.65:33233: connect: Connection refused (error 111)
I20260806 17:03:20.027024 20749 ts_tablet_manager.cc:1918] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:03:20.027181 20749 log.cc:1196] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/b1ddf0678a1e480d834e0a374fa8caf3
I20260806 17:03:20.027539 20749 ts_tablet_manager.cc:1939] T b1ddf0678a1e480d834e0a374fa8caf3 P 7c24feac5d084c9aa5cd8f931399a478: Deleting consensus metadata
I20260806 17:03:20.029429 20212 catalog_manager.cc:4841] TS 7c24feac5d084c9aa5cd8f931399a478 (127.9.111.67:36135): tablet b1ddf0678a1e480d834e0a374fa8caf3 (table test-table [id=f38cf31f8a33404fbc65cbdb2d89b221]) successfully deleted
I20260806 17:03:20.031488 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 20568
I20260806 17:03:20.038439 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 20194
2026-08-06T17:03:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2851 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:03:20 dist-test-slave-69rg krb5kdc[20755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:03:20 dist-test-slave-69rg krb5kdc[20755](info): set up 2 sockets
Aug 06 17:03:20 dist-test-slave-69rg krb5kdc[20755](info): commencing operation
krb5kdc: starting...
W20260806 17:03:21.206319 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:03:21 dist-test-slave-69rg krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:03:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:03:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.111.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.111.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.111.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/rangeradmin_127.9.111.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.111.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/rangeradmin_127.9.111.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.111.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.111.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.111.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/rangerlookup_127.9.111.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.111.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/rangerlookup_127.9.111.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.111.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.111.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.111.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/HTTP_127.9.111.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.111.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/HTTP_127.9.111.84@KRBTEST.COM.keytab.
I20260806 17:03:21.444469 9661 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7cxihq/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/postgres -l logfile start
2026-08-06 17:03:24.399 UTC [20785] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 17:03:24.399 UTC [20785] LOG: listening on IPv4 address "127.9.111.84", port 36191
2026-08-06 17:03:24.404 UTC [20785] LOG: listening on Unix socket "/tmp/.s.PGSQL.36191"
2026-08-06 17:03:24.427 UTC [20788] LOG: database system was shut down at 2026-08-06 17:03:22 UTC
2026-08-06 17:03:24.432 UTC [20785] LOG: database system is ready to accept connections
I20260806 17:03:25.407953 9661 mini_postgres.cc:96] Postgres bound to 36191
2026-08-06 17:03:25.417 UTC [20796] FATAL: database "slave" does not exist
127.9.111.84:36191 - accepting connections
I20260806 17:03:25.418332 9661 mini_ranger.cc:161] Starting Ranger...
I20260806 17:03:25.437390 9661 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 17:03:26.038833 9661 mini_ranger.cc:87] Created ranger Postgres database
I20260806 17:03:26.038919 9661 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ranger-admin
2026-08-06 17:03:26,380 [I] DB FLAVOR :POSTGRES
2026-08-06 17:03:26,380 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 17:03:26,380 [I] Checking connection..
2026-08-06 17:03:26,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 17:03:26,659 [I] Checking connection passed.
2026-08-06 17:03:26,659 [I] --------- Verifying version history table ---------
2026-08-06 17:03:26,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 17:03:26,940 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 17:03:26,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 17:03:27,188 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 17:03:27,189 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 17:03:27,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 17:03:27.444 UTC [20860] WARNING: there is no transaction in progress
2026-08-06 17:03:27,452 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 17:03:27,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 17:03:27,711 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 17:03:27,712 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 17:03:27,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 17:03:27,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 17:03:28,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-69rg.c.gcp-upstream.internal','N') ;"
2026-08-06 17:03:28,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 17:03:28,740 [I] Table x_portal_user does not exist in database ranger
2026-08-06 17:03:28,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 17:03:29,009 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 17:03:29,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 17:03:29,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 17:03:29,476 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 17:03:29,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 17:03:30.516 UTC [20974] WARNING: there is no transaction in progress
2026-08-06 17:03:30.535 UTC [20974] WARNING: there is no transaction in progress
2026-08-06 17:03:30.554 UTC [20974] WARNING: there is no transaction in progress
2026-08-06 17:03:30.573 UTC [20974] WARNING: there is no transaction in progress
2026-08-06 17:03:30.595 UTC [20974] WARNING: there is no transaction in progress
2026-08-06 17:03:30.618 UTC [20974] WARNING: there is no transaction in progress
2026-08-06 17:03:30.687 UTC [20974] WARNING: there is no transaction in progress
2026-08-06 17:03:30.696 UTC [20974] WARNING: there is no transaction in progress
2026-08-06 17:03:30.712 UTC [20974] WARNING: there is no transaction in progress
2026-08-06 17:03:30.723 UTC [20974] WARNING: there is no transaction in progress
2026-08-06 17:03:30.735 UTC [20974] WARNING: there is no transaction in progress
2026-08-06 17:03:30.751 UTC [20974] WARNING: there is no transaction in progress
2026-08-06 17:03:30.762 UTC [20974] WARNING: there is no transaction in progress
2026-08-06 17:03:30.769 UTC [20974] WARNING: there is no transaction in progress
2026-08-06 17:03:30.777 UTC [20974] WARNING: there is no transaction in progress
2026-08-06 17:03:30.785 UTC [20974] WARNING: there is no transaction in progress
2026-08-06 17:03:31,634 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 17:03:31,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 17:03:31,901 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 17:03:31,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 17:03:32,150 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 17:03:32,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 17:03:32,415 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 17:03:32,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 17:03:32,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 17:03:32,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-69rg.c.gcp-upstream.internal';"
2026-08-06 17:03:33,171 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 17:03:33,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 17:03:33,441 [I] DB_PATCHES have already been applied
I20260806 17:03:33.447069 9661 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ranger-admin:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7cxihq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 17:03:33.447247 9661 mini_ranger.cc:193] Using host: 127.9.111.84
I20260806 17:03:33.451663 9661 mini_ranger.cc:227] Ranger admin URL: http://127.9.111.84:46315
Aug 06, 2026 5:03:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 5:03:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 5:03:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 5:03:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 5:03:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 5:03:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 5:03:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 17:03:34.615705 9661 mini_ranger.cc:160] Time spent starting Ranger: real 9.197s user 0.003s sys 0.010s
Aug 06, 2026 5:03:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 5:03:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 06, 2026 5:03:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 5:03:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 5:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 5:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 5:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 5:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 5:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 5:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 5:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 5:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 5:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 5:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 5:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 5:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 5:03:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 5:03:56 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 17:03:57.174028 9661 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.111.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.111.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.111.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/rangerkms_127.9.111.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.111.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/rangerkms_127.9.111.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.111.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.111.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.111.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/HTTP_127.9.111.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.111.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/HTTP_127.9.111.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 17:03:57 dist-test-slave-69rg krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035837, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 17:03:57.276337 9661 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 17:03:57.288326 9661 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 17:03:58.327346 9661 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 17:03:58.327473 9661 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ranger-kms
I20260806 17:03:58.327602 9661 mini_ranger_kms.cc:227] Using postgres at 127.9.111.84:36191
2026-08-06 17:03:58,784 [I] DB FLAVOR :POSTGRES
2026-08-06 17:03:58,784 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 17:03:58,784 [I] Checking connection
2026-08-06 17:03:58,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 17:03:59,082 [I] connection success
2026-08-06 17:03:59,082 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 17:03:59,082 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 17:03:59,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 17:03:59,373 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 17:03:59,373 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 17:03:59,374 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 17:03:59,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:36191/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 17:03:59,664 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 17:04:00.173734 9661 mini_ranger_kms.cc:310] Created kms service
I20260806 17:04:00.426834 9661 mini_ranger_kms.cc:326] Created kudu user
I20260806 17:04:00.477407 9661 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 17:04:00 dist-test-slave-69rg krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035837, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
I20260806 17:04:00.948576 9661 mini_ranger_kms.cc:384] Added ranger policy
I20260806 17:04:00.949079 9661 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ranger-kms:/tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 17:04:00.949142 9661 mini_ranger_kms.cc:240] Using host: 127.9.111.84
I20260806 17:04:00.955210 9661 mini_ranger_kms.cc:276] Ranger KMS PID: 21167
I20260806 17:04:00.955402 9661 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.111.84:39097
Aug 06, 2026 5:04:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 5:04:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 5:04:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 5:04:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 5:04:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 5:04:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260806 17:04:03.277324 9661 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.001s user 0.000s sys 0.438s
I20260806 17:04:03.277462 9661 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 17:04:03.277577 9661 mini_ranger_kms.cc:401] 127.9.111.84:39097/kms/v1/keys
Aug 06 17:04:03 dist-test-slave-69rg krb5kdc[20755](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786035843, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.111.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 17:04:04 dist-test-slave-69rg krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035843, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.111.84@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
Aug 06 17:04:04 dist-test-slave-69rg krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035843, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.111.84@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
Aug 06 17:04:06 dist-test-slave-69rg krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:04:06.758689 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:35203
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:35203
--ranger_config_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.111.84:39097/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:04:06.901571 21222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:04:06.901926 21222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:04:06.902016 21222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:04:06.906479 21222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:04:06.906550 21222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:04:06.906570 21222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:04:06.906587 21222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:04:06.906605 21222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:04:06.912540 21222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.111.84:39097/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:35203
--ranger_config_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:35203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:04:06.914103 21222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:04:06.915283 21222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:04:06.926920 21230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:04:06.927136 21222 server_base.cc:1034] running on GCE node
W20260806 17:04:06.927074 21227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:04:06.926959 21228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:04:06.928045 21222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:04:06.929244 21222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:04:06.930423 21222 hybrid_clock.cc:648] HybridClock initialized: now 1786035846930394 us; error 48 us; skew 500 ppm
Aug 06 17:04:06 dist-test-slave-69rg krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035846, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:04:06.934012 21222 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:04:06.935037 21222 webserver.cc:460] Webserver started at http://127.9.111.126:35485/ using document root <none> and password file <none>
I20260806 17:04:06.935617 21222 fs_manager.cc:359] Metadata directory not provided
I20260806 17:04:06.935703 21222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:04:06.935922 21222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 17:04:06 dist-test-slave-69rg krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035846, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
I20260806 17:04:06.986722 21222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "51b5baf21ef6430380ad8e2acfb8d221"
format_stamp: "Formatted at 2026-08-06 17:04:06 on dist-test-slave-69rg"
server_key: "64d2e9172653784e849fd773d15da6a8"
server_key_iv: "ecd2788596e9e35c9d4f4b64379cdfb1"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:06.987558 21222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "51b5baf21ef6430380ad8e2acfb8d221"
format_stamp: "Formatted at 2026-08-06 17:04:06 on dist-test-slave-69rg"
server_key: "64d2e9172653784e849fd773d15da6a8"
server_key_iv: "ecd2788596e9e35c9d4f4b64379cdfb1"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:06.991798 21222 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260806 17:04:07.006666 21238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:04:07.008438 21222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 17:04:07.008631 21222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "51b5baf21ef6430380ad8e2acfb8d221"
format_stamp: "Formatted at 2026-08-06 17:04:06 on dist-test-slave-69rg"
server_key: "64d2e9172653784e849fd773d15da6a8"
server_key_iv: "ecd2788596e9e35c9d4f4b64379cdfb1"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:07.008812 21222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:04:07.026437 21222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:04:07.029646 21222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:04:07.029922 21222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:04:07.038940 21222 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:35203
I20260806 17:04:07.038998 21296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:35203 every 8 connection(s)
I20260806 17:04:07.040055 21222 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:04:07.043654 21297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:04:07.048774 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 21222
I20260806 17:04:07.048936 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:04:07.049237 9661 external_mini_cluster.cc:1428] Setting key 4ef8c33d0c795264aeb5fd59fb778c82
I20260806 17:04:07.051420 21297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221: Bootstrap starting.
I20260806 17:04:07.055369 21297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221: Neither blocks nor log segments found. Creating new log.
I20260806 17:04:07.056567 21297 log.cc:824] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221: Log is configured to *not* fsync() on all Append() calls
Aug 06 17:04:07 dist-test-slave-69rg krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:04:07.059504 21297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221: No bootstrap required, opened a new log
I20260806 17:04:07.064468 21297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b5baf21ef6430380ad8e2acfb8d221" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 35203 } }
I20260806 17:04:07.064781 21297 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:04:07.064873 21297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b5baf21ef6430380ad8e2acfb8d221, State: Initialized, Role: FOLLOWER
I20260806 17:04:07.065385 21297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b5baf21ef6430380ad8e2acfb8d221" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 35203 } }
I20260806 17:04:07.065559 21297 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:04:07.065656 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:07.050975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46802 (local address 127.9.111.126:35203)
0806 17:04:07.051415 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0806 17:04:07.051437 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 17:04:07.051475 (+ 38us) server_negotiation.cc:407] Connection header received
0806 17:04:07.052298 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:04:07.052326 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:04:07.052677 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:04:07.053035 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:04:07.053926 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:07.054986 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:07.055961 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:07.056447 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:07.056632 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:07.059926 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:04:07.059935 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:04:07.059955 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:04:07.060000 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:04:07.062177 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:04:07.062749 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:04:07.062756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:04:07.062762 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:04:07.062906 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:04:07.063242 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:04:07.063245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:04:07.063247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:04:07.063837 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:04:07.064047 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 17:04:07.064358 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 17:04:07.064665 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":141,"threads_started":1}
I20260806 17:04:07.065667 21297 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:04:07.065806 21297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:04:07.067267 21297 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b5baf21ef6430380ad8e2acfb8d221" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 35203 } }
I20260806 17:04:07.067649 21297 leader_election.cc:302] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51b5baf21ef6430380ad8e2acfb8d221; no voters:
I20260806 17:04:07.068014 21297 leader_election.cc:288] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:04:07.068370 21302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:04:07.068619 21302 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221 [term 1 LEADER]: Becoming Leader. State: Replica: 51b5baf21ef6430380ad8e2acfb8d221, State: Running, Role: LEADER
I20260806 17:04:07.068995 21302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b5baf21ef6430380ad8e2acfb8d221" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 35203 } }
I20260806 17:04:07.069473 21297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:04:07.070216 21302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51b5baf21ef6430380ad8e2acfb8d221. Latest consensus state: current_term: 1 leader_uuid: "51b5baf21ef6430380ad8e2acfb8d221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b5baf21ef6430380ad8e2acfb8d221" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 35203 } } }
I20260806 17:04:07.070415 21302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221 [sys.catalog]: This master's current role is: LEADER
I20260806 17:04:07.070688 21304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51b5baf21ef6430380ad8e2acfb8d221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b5baf21ef6430380ad8e2acfb8d221" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 35203 } } }
I20260806 17:04:07.070829 21310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:04:07.070865 21304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221 [sys.catalog]: This master's current role is: LEADER
I20260806 17:04:07.074635 21310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:04:07.075201 21297 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 17:04:07.086169 21310 catalog_manager.cc:1347] Generated new cluster ID: 5e9263db9cc74d578798e03d76da1b32
I20260806 17:04:07.086364 21310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:04:07.097051 21310 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:04:07.099839 21310 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:04:07.109623 21310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221: Generated new TSK 0
I20260806 17:04:07.110698 21310 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
I20260806 17:04:07.914194 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.111.84:39097/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:35203
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:04:08.084120 21342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:04:08.084467 21342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:04:08.084570 21342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:04:08.089879 21342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:04:08.090016 21342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:04:08.090128 21342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:04:08.096074 21342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.111.84:39097/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:35203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:04:08.097275 21342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:04:08.098459 21342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:04:08.108896 21348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:04:08.109035 21347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:04:08.108909 21350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:04:08.110113 21342 server_base.cc:1034] running on GCE node
I20260806 17:04:08.110630 21342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:04:08.111449 21342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:04:08.112636 21342 hybrid_clock.cc:648] HybridClock initialized: now 1786035848112607 us; error 39 us; skew 500 ppm
Aug 06 17:04:08 dist-test-slave-69rg krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035848, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:04:08.116487 21342 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:04:08.117699 21342 webserver.cc:460] Webserver started at http://127.9.111.65:42749/ using document root <none> and password file <none>
I20260806 17:04:08.118355 21342 fs_manager.cc:359] Metadata directory not provided
I20260806 17:04:08.118420 21342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:04:08.118666 21342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 17:04:08 dist-test-slave-69rg krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035848, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
I20260806 17:04:08.136025 21342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "a6f4eefe2b9948978263c01fd300bb7c"
format_stamp: "Formatted at 2026-08-06 17:04:08 on dist-test-slave-69rg"
server_key: "d1d8c29679c527028fd944e93e5d6584"
server_key_iv: "3f150e85e040d4492817fd1d5f24fe50"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:08.136870 21342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "a6f4eefe2b9948978263c01fd300bb7c"
format_stamp: "Formatted at 2026-08-06 17:04:08 on dist-test-slave-69rg"
server_key: "d1d8c29679c527028fd944e93e5d6584"
server_key_iv: "3f150e85e040d4492817fd1d5f24fe50"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:08.143404 21342 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 17:04:08.162158 21359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:04:08.163941 21342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:04:08.164098 21342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "a6f4eefe2b9948978263c01fd300bb7c"
format_stamp: "Formatted at 2026-08-06 17:04:08 on dist-test-slave-69rg"
server_key: "d1d8c29679c527028fd944e93e5d6584"
server_key_iv: "3f150e85e040d4492817fd1d5f24fe50"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:08.164278 21342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:04:08.187408 21342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:04:08.190690 21342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:04:08.190908 21342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:04:08.191598 21342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:04:08.192655 21342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:04:08.192708 21342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:04:08.192760 21342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:04:08.192781 21342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:04:08.208024 21342 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:42371
I20260806 17:04:08.209270 21342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:04:08.211165 21474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:42371 every 8 connection(s)
I20260806 17:04:08.218504 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 21342
I20260806 17:04:08.218657 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:04:08.219038 9661 external_mini_cluster.cc:1428] Setting key fbf2e8bc53ef0d28a5f36ec314774fae
Aug 06 17:04:08 dist-test-slave-69rg krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035848, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:04:08.242241 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:08.214094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:35203 (local address 127.9.111.65:50735)
0806 17:04:08.215430 (+ 1336us) negotiation.cc:107] Waiting for socket to connect
0806 17:04:08.215491 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 17:04:08.216673 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:04:08.226886 (+ 10213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:04:08.226900 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:04:08.227340 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:04:08.228111 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:08.228137 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:08.229424 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:08.229428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:08.231001 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:08.231014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:08.231840 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:08.231847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:08.232139 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:08.233473 (+ 1334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:04:08.233500 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:04:08.235504 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:04:08.238389 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:04:08.238398 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:04:08.238419 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:04:08.238857 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:04:08.239268 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:04:08.239271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:04:08.239273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:04:08.239533 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:04:08.240099 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:04:08.240107 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:04:08.240488 (+ 381us) client_negotiation.cc:769] Sending connection context
0806 17:04:08.240700 (+ 212us) client_negotiation.cc:240] Negotiation successful
0806 17:04:08.241094 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"thread_start_us":139,"threads_started":1}
I20260806 17:04:08.243206 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:08.213480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:50735 (local address 127.9.111.126:35203)
0806 17:04:08.226150 (+ 12670us) server_negotiation.cc:206] Beginning negotiation
0806 17:04:08.226155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:04:08.226179 (+ 24us) server_negotiation.cc:407] Connection header received
0806 17:04:08.226237 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:04:08.226240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:04:08.226301 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:04:08.226425 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:04:08.228336 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:08.229250 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:08.231163 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:08.231541 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:08.231664 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:08.235800 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:04:08.235804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:04:08.235808 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:04:08.235845 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:04:08.238196 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:04:08.239002 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:04:08.239007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:04:08.239009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:04:08.239113 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:04:08.239663 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:04:08.239667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:04:08.239669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:04:08.239941 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:04:08.240039 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 17:04:08.242862 (+ 2823us) server_negotiation.cc:299] Negotiation successful
0806 17:04:08.243019 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12511,"thread_start_us":102,"threads_started":1}
I20260806 17:04:08.245189 21475 heartbeater.cc:347] Connected to a master server at 127.9.111.126:35203
I20260806 17:04:08.245464 21475 heartbeater.cc:464] Registering TS with master...
I20260806 17:04:08.246131 21475 heartbeater.cc:511] Master 127.9.111.126:35203 requested a full tablet report, sending...
I20260806 17:04:08.248323 21257 ts_manager.cc:194] Registered new tserver with Master: a6f4eefe2b9948978263c01fd300bb7c (127.9.111.65:42371)
I20260806 17:04:08.250398 21257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:50735
WARNING: no policy specified for kudu/127.9.111.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
I20260806 17:04:08.291177 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.111.84:39097/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:35203
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 17:04:08 dist-test-slave-69rg krb5kdc[20755](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786035848, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 17:04:08.443295 21483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:04:08.443608 21483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:04:08.443675 21483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:04:08.449076 21483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:04:08.449190 21483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:04:08.449314 21483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:04:08.455525 21483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.111.84:39097/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:35203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:04:08.456755 21483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:04:08.457883 21483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:04:08.469022 21491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:04:08.469022 21493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:04:08.469502 21483 server_base.cc:1034] running on GCE node
W20260806 17:04:08.469748 21489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:04:08.470260 21483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:04:08.471186 21483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:04:08.472424 21483 hybrid_clock.cc:648] HybridClock initialized: now 1786035848472377 us; error 90 us; skew 500 ppm
Aug 06 17:04:08 dist-test-slave-69rg krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035848, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:04:08.479897 21483 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:04:08.480991 21483 webserver.cc:460] Webserver started at http://127.9.111.66:44027/ using document root <none> and password file <none>
I20260806 17:04:08.481647 21483 fs_manager.cc:359] Metadata directory not provided
I20260806 17:04:08.481751 21483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:04:08.481998 21483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 17:04:08 dist-test-slave-69rg krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035848, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
I20260806 17:04:08.504371 21483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "eced2e1212f042cfb32e97f349322ed0"
format_stamp: "Formatted at 2026-08-06 17:04:08 on dist-test-slave-69rg"
server_key: "9c98e0b05834086180cff3dcf8b540a3"
server_key_iv: "0a84b1473350c2bd44470ca09f0611f7"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:08.505064 21483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "eced2e1212f042cfb32e97f349322ed0"
format_stamp: "Formatted at 2026-08-06 17:04:08 on dist-test-slave-69rg"
server_key: "9c98e0b05834086180cff3dcf8b540a3"
server_key_iv: "0a84b1473350c2bd44470ca09f0611f7"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:08.510419 21483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 17:04:08.531169 21500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:04:08.532831 21483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:04:08.532990 21483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "eced2e1212f042cfb32e97f349322ed0"
format_stamp: "Formatted at 2026-08-06 17:04:08 on dist-test-slave-69rg"
server_key: "9c98e0b05834086180cff3dcf8b540a3"
server_key_iv: "0a84b1473350c2bd44470ca09f0611f7"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:08.533144 21483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:04:08.565220 21483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:04:08.569494 21483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:04:08.569718 21483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:04:08.570719 21483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:04:08.572006 21483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:04:08.572079 21483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:04:08.572157 21483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:04:08.572190 21483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:04:08.584235 21483 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:36253
I20260806 17:04:08.585487 21483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:04:08.588953 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 21483
I20260806 17:04:08.589099 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:04:08.589457 9661 external_mini_cluster.cc:1428] Setting key b6b2ca9a721e224baae5d9f6d29f6a89
I20260806 17:04:08.593891 21620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:36253 every 8 connection(s)
WARNING: no policy specified for kudu/127.9.111.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.67@KRBTEST.COM" created.
Aug 06 17:04:08 dist-test-slave-69rg krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035848, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:04:08.624120 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:08.597307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:50951 (local address 127.9.111.126:35203)
0806 17:04:08.597460 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 17:04:08.597466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:04:08.605910 (+ 8444us) server_negotiation.cc:407] Connection header received
0806 17:04:08.607899 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:04:08.607920 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:04:08.607985 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:04:08.608045 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:04:08.613091 (+ 5046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:08.613963 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:08.615879 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:08.616225 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:08.616312 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:08.619651 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:04:08.619654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:04:08.619658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:04:08.619689 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:04:08.621522 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:04:08.622210 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:04:08.622214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:04:08.622215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:04:08.622305 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:04:08.622712 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:04:08.622716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:04:08.622718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:04:08.622956 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:04:08.623049 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 17:04:08.623806 (+ 757us) server_negotiation.cc:299] Negotiation successful
0806 17:04:08.623948 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 17:04:08.626129 21624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:08.596875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:35203 (local address 127.9.111.66:50951)
0806 17:04:08.605629 (+ 8754us) negotiation.cc:107] Waiting for socket to connect
0806 17:04:08.605717 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0806 17:04:08.607579 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:04:08.608800 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:04:08.608812 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:04:08.609213 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:04:08.612875 (+ 3662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:08.612903 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:08.614171 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:08.614174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:08.615709 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:08.615722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:08.616549 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:08.616553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:08.616760 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:08.617508 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:04:08.617532 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:04:08.619434 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:04:08.621674 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:04:08.621680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:04:08.621699 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:04:08.622087 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:04:08.622418 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:04:08.622421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:04:08.622424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:04:08.622600 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:04:08.623139 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:04:08.623146 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:04:08.623501 (+ 355us) client_negotiation.cc:769] Sending connection context
0806 17:04:08.624608 (+ 1107us) client_negotiation.cc:240] Negotiation successful
0806 17:04:08.624978 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8442,"thread_start_us":152,"threads_started":1}
I20260806 17:04:08.631098 21621 heartbeater.cc:347] Connected to a master server at 127.9.111.126:35203
I20260806 17:04:08.631352 21621 heartbeater.cc:464] Registering TS with master...
I20260806 17:04:08.631999 21621 heartbeater.cc:511] Master 127.9.111.126:35203 requested a full tablet report, sending...
I20260806 17:04:08.633217 21257 ts_manager.cc:194] Registered new tserver with Master: eced2e1212f042cfb32e97f349322ed0 (127.9.111.66:36253)
I20260806 17:04:08.634115 21257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:50951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
I20260806 17:04:08.651542 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.111.84:39097/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:35203
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:04:08.839273 21626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:04:08.839828 21626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:04:08.839910 21626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:04:08.846315 21626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:04:08.846427 21626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:04:08.846517 21626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:04:08.854803 21626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.111.84:39097/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:35203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:04:08.856280 21626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:04:08.857617 21626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:04:08.873697 21631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:04:08.874521 21634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:04:08.875077 21626 server_base.cc:1034] running on GCE node
W20260806 17:04:08.875366 21632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:04:08.875938 21626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:04:08.877055 21626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:04:08.878302 21626 hybrid_clock.cc:648] HybridClock initialized: now 1786035848878229 us; error 91 us; skew 500 ppm
Aug 06 17:04:08 dist-test-slave-69rg krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035848, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:04:08.882612 21626 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:04:08.884119 21626 webserver.cc:460] Webserver started at http://127.9.111.67:43217/ using document root <none> and password file <none>
I20260806 17:04:08.884928 21626 fs_manager.cc:359] Metadata directory not provided
I20260806 17:04:08.885087 21626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:04:08.885377 21626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 17:04:08 dist-test-slave-69rg krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035848, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
I20260806 17:04:08.916196 21626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "ab3ff57c85b043018039580e8509c746"
format_stamp: "Formatted at 2026-08-06 17:04:08 on dist-test-slave-69rg"
server_key: "8582c6828c8fc676856c8448c8684cff"
server_key_iv: "23d44273b1f3a5d3bd4b4e7bbe73f65f"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:08.917797 21626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "ab3ff57c85b043018039580e8509c746"
format_stamp: "Formatted at 2026-08-06 17:04:08 on dist-test-slave-69rg"
server_key: "8582c6828c8fc676856c8448c8684cff"
server_key_iv: "23d44273b1f3a5d3bd4b4e7bbe73f65f"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:08.923780 21626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 17:04:08.946645 21641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
Aug 06 17:04:08 dist-test-slave-69rg krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035848, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
I20260806 17:04:08.949178 21626 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260806 17:04:08.949338 21626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "ab3ff57c85b043018039580e8509c746"
format_stamp: "Formatted at 2026-08-06 17:04:08 on dist-test-slave-69rg"
server_key: "8582c6828c8fc676856c8448c8684cff"
server_key_iv: "23d44273b1f3a5d3bd4b4e7bbe73f65f"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:08.949493 21626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:04:08.978034 21626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:04:08.982028 21626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:04:08.982239 21626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:04:08.982897 21626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:04:08.983928 21626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:04:08.983994 21626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:04:08.984048 21626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:04:08.984078 21626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:04:09.000157 21626 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:43427
I20260806 17:04:09.001545 21626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:04:09.006424 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 21626
I20260806 17:04:09.006552 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:04:09.006860 9661 external_mini_cluster.cc:1428] Setting key afa8eca8a6a5ec5caf46ae62e24266d5
I20260806 17:04:09.006032 21757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:43427 every 8 connection(s)
Aug 06 17:04:09 dist-test-slave-69rg krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035848, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:04:09.064953 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:09.005345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:45411 (local address 127.9.111.126:35203)
0806 17:04:09.007284 (+ 1939us) server_negotiation.cc:206] Beginning negotiation
0806 17:04:09.007291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:04:09.016372 (+ 9081us) server_negotiation.cc:407] Connection header received
0806 17:04:09.024170 (+ 7798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:04:09.024174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:04:09.024247 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:04:09.024304 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:04:09.026916 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.027780 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.055122 (+ 27342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.055585 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.055705 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:09.059655 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:04:09.059660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:04:09.059663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:04:09.059695 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:04:09.061954 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:04:09.062814 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:04:09.062818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:04:09.062820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:04:09.062923 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:04:09.063422 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:04:09.063426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:04:09.063427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:04:09.063691 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:04:09.063789 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 17:04:09.064615 (+ 826us) server_negotiation.cc:299] Negotiation successful
0806 17:04:09.064776 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1821}
I20260806 17:04:09.067118 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:09.005956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:35203 (local address 127.9.111.67:45411)
0806 17:04:09.016153 (+ 10197us) negotiation.cc:107] Waiting for socket to connect
0806 17:04:09.016251 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0806 17:04:09.018485 (+ 2234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:04:09.024881 (+ 6396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:04:09.024893 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:04:09.025361 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:04:09.026160 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:09.026182 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.042933 (+ 16751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.042938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:09.044786 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:09.044809 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.056016 (+ 11207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.056021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:09.056359 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:09.057184 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:04:09.057212 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:04:09.059417 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:04:09.062169 (+ 2752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:04:09.062178 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:04:09.062201 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:04:09.062668 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:04:09.063038 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:04:09.063042 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:04:09.063044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:04:09.063294 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:04:09.063873 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:04:09.063882 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:04:09.064289 (+ 407us) client_negotiation.cc:769] Sending connection context
0806 17:04:09.065508 (+ 1219us) client_negotiation.cc:240] Negotiation successful
0806 17:04:09.065954 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9847,"thread_start_us":298,"threads_started":1}
I20260806 17:04:09.068850 21758 heartbeater.cc:347] Connected to a master server at 127.9.111.126:35203
I20260806 17:04:09.069106 21758 heartbeater.cc:464] Registering TS with master...
I20260806 17:04:09.069754 21758 heartbeater.cc:511] Master 127.9.111.126:35203 requested a full tablet report, sending...
I20260806 17:04:09.072434 21257 ts_manager.cc:194] Registered new tserver with Master: ab3ff57c85b043018039580e8509c746 (127.9.111.67:43427)
I20260806 17:04:09.073243 21257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:45411
I20260806 17:04:09.083863 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 17:04:09.098608 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:09.087007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46818 (local address 127.9.111.126:35203)
0806 17:04:09.087174 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 17:04:09.087179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:04:09.087680 (+ 501us) server_negotiation.cc:407] Connection header received
0806 17:04:09.087886 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:04:09.087890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:04:09.087943 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:04:09.087998 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:04:09.089148 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.090119 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.091562 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.091933 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.092019 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:09.094354 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:04:09.094359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:04:09.094362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:04:09.094396 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:04:09.096419 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:04:09.096992 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:04:09.096996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:04:09.096999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:04:09.097093 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:04:09.097451 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:04:09.097455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:04:09.097456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:04:09.097711 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:04:09.097847 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 17:04:09.098290 (+ 443us) server_negotiation.cc:299] Negotiation successful
0806 17:04:09.098429 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 17:04:09.102824 21257 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:04:09.107712 21257 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:04:09.131227 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:09.122081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:42371 (local address 127.0.0.1:52790)
0806 17:04:09.124341 (+ 2260us) negotiation.cc:107] Waiting for socket to connect
0806 17:04:09.124406 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 17:04:09.124818 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:04:09.125703 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:04:09.125709 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:04:09.125734 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:04:09.126058 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:09.126070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.127553 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.127559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:09.129135 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:09.129145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.130610 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.130615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:09.130756 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:09.130807 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 17:04:09.130929 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 17:04:09.131052 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2048,"thread_start_us":149,"threads_started":1}
I20260806 17:04:09.131980 21772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:09.122231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52790 (local address 127.9.111.65:42371)
0806 17:04:09.124675 (+ 2444us) server_negotiation.cc:206] Beginning negotiation
0806 17:04:09.124684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 17:04:09.124712 (+ 28us) server_negotiation.cc:407] Connection header received
0806 17:04:09.125057 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:04:09.125064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:04:09.125324 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:04:09.125484 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:04:09.126216 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.127405 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.129280 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.130241 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.130356 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:09.130529 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 17:04:09.131701 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0806 17:04:09.131823 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2310,"thread_start_us":87,"threads_started":1}
I20260806 17:04:09.134806 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:09.123070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:36253 (local address 127.0.0.1:33170)
0806 17:04:09.125135 (+ 2065us) negotiation.cc:107] Waiting for socket to connect
0806 17:04:09.125150 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:04:09.125240 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:04:09.126406 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:04:09.126409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:04:09.126428 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:04:09.127432 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:09.127442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.129018 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.129021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:09.131252 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:09.131266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.134347 (+ 3081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.134351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:09.134455 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:09.134481 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 17:04:09.134561 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 17:04:09.134660 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1986,"thread_start_us":143,"threads_started":1}
I20260806 17:04:09.135285 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:09.123761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33170 (local address 127.9.111.66:36253)
0806 17:04:09.125493 (+ 1732us) server_negotiation.cc:206] Beginning negotiation
0806 17:04:09.125510 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 17:04:09.125533 (+ 23us) server_negotiation.cc:407] Connection header received
0806 17:04:09.125612 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:04:09.125618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:04:09.125811 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:04:09.126254 (+ 443us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:04:09.127594 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.128863 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.133069 (+ 4206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.133998 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.134119 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:09.134275 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 17:04:09.135013 (+ 738us) server_negotiation.cc:299] Negotiation successful
0806 17:04:09.135130 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1617,"thread_start_us":90,"threads_started":1}
I20260806 17:04:09.135627 21409 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7496873b1744a6b43446b2cdaf5940 (DEFAULT_TABLE table=test-table [id=89ad3013be9543a8b606977f74cf781e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:04:09.137264 21409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7496873b1744a6b43446b2cdaf5940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:04:09.138446 21532 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7496873b1744a6b43446b2cdaf5940 (DEFAULT_TABLE table=test-table [id=89ad3013be9543a8b606977f74cf781e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:04:09.139549 21532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7496873b1744a6b43446b2cdaf5940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:04:09.143199 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:09.123664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:43427 (local address 127.0.0.1:40662)
0806 17:04:09.131446 (+ 7782us) negotiation.cc:107] Waiting for socket to connect
0806 17:04:09.131461 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:04:09.131548 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:04:09.132478 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:04:09.132481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:04:09.132499 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:04:09.132577 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:09.132582 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.139090 (+ 6508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.139096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:09.140564 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:09.140579 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.142698 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.142704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:09.142807 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:09.142834 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 17:04:09.142921 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 17:04:09.143012 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7718,"thread_start_us":96,"threads_started":1}
I20260806 17:04:09.143790 21776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:09.123863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.9.111.67:43427)
0806 17:04:09.124963 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0806 17:04:09.124980 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 17:04:09.131597 (+ 6617us) server_negotiation.cc:407] Connection header received
0806 17:04:09.131704 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:04:09.131712 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:04:09.131966 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:04:09.132175 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:04:09.132718 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.138813 (+ 6095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.140783 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.141746 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.142440 (+ 694us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:09.142624 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 17:04:09.143494 (+ 870us) server_negotiation.cc:299] Negotiation successful
0806 17:04:09.143629 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":980,"thread_start_us":142,"threads_started":1}
I20260806 17:04:09.147469 21691 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7496873b1744a6b43446b2cdaf5940 (DEFAULT_TABLE table=test-table [id=89ad3013be9543a8b606977f74cf781e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:04:09.148862 21779 tablet_bootstrap.cc:492] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c: Bootstrap starting.
I20260806 17:04:09.148986 21691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7496873b1744a6b43446b2cdaf5940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:04:09.149896 21778 tablet_bootstrap.cc:492] T 3e7496873b1744a6b43446b2cdaf5940 P eced2e1212f042cfb32e97f349322ed0: Bootstrap starting.
I20260806 17:04:09.151780 21779 tablet_bootstrap.cc:654] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c: Neither blocks nor log segments found. Creating new log.
I20260806 17:04:09.152277 21778 tablet_bootstrap.cc:654] T 3e7496873b1744a6b43446b2cdaf5940 P eced2e1212f042cfb32e97f349322ed0: Neither blocks nor log segments found. Creating new log.
I20260806 17:04:09.153231 21779 log.cc:824] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c: Log is configured to *not* fsync() on all Append() calls
I20260806 17:04:09.155740 21778 log.cc:824] T 3e7496873b1744a6b43446b2cdaf5940 P eced2e1212f042cfb32e97f349322ed0: Log is configured to *not* fsync() on all Append() calls
I20260806 17:04:09.159425 21778 tablet_bootstrap.cc:492] T 3e7496873b1744a6b43446b2cdaf5940 P eced2e1212f042cfb32e97f349322ed0: No bootstrap required, opened a new log
I20260806 17:04:09.159766 21778 ts_tablet_manager.cc:1397] T 3e7496873b1744a6b43446b2cdaf5940 P eced2e1212f042cfb32e97f349322ed0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260806 17:04:09.160714 21782 tablet_bootstrap.cc:492] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746: Bootstrap starting.
I20260806 17:04:09.162182 21779 tablet_bootstrap.cc:492] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c: No bootstrap required, opened a new log
I20260806 17:04:09.162406 21779 ts_tablet_manager.cc:1397] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260806 17:04:09.163861 21782 tablet_bootstrap.cc:654] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746: Neither blocks nor log segments found. Creating new log.
I20260806 17:04:09.164798 21782 log.cc:824] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746: Log is configured to *not* fsync() on all Append() calls
I20260806 17:04:09.168252 21779 raft_consensus.cc:348] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f4eefe2b9948978263c01fd300bb7c" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42371 } } peers { permanent_uuid: "eced2e1212f042cfb32e97f349322ed0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36253 } } peers { permanent_uuid: "ab3ff57c85b043018039580e8509c746" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43427 } }
I20260806 17:04:09.168679 21779 raft_consensus.cc:374] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:04:09.168805 21779 raft_consensus.cc:729] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f4eefe2b9948978263c01fd300bb7c, State: Initialized, Role: FOLLOWER
I20260806 17:04:09.169458 21779 consensus_queue.cc:260] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f4eefe2b9948978263c01fd300bb7c" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42371 } } peers { permanent_uuid: "eced2e1212f042cfb32e97f349322ed0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36253 } } peers { permanent_uuid: "ab3ff57c85b043018039580e8509c746" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43427 } }
I20260806 17:04:09.170501 21778 raft_consensus.cc:348] T 3e7496873b1744a6b43446b2cdaf5940 P eced2e1212f042cfb32e97f349322ed0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f4eefe2b9948978263c01fd300bb7c" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42371 } } peers { permanent_uuid: "eced2e1212f042cfb32e97f349322ed0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36253 } } peers { permanent_uuid: "ab3ff57c85b043018039580e8509c746" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43427 } }
I20260806 17:04:09.170785 21778 raft_consensus.cc:374] T 3e7496873b1744a6b43446b2cdaf5940 P eced2e1212f042cfb32e97f349322ed0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:04:09.170849 21778 raft_consensus.cc:729] T 3e7496873b1744a6b43446b2cdaf5940 P eced2e1212f042cfb32e97f349322ed0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eced2e1212f042cfb32e97f349322ed0, State: Initialized, Role: FOLLOWER
I20260806 17:04:09.171120 21779 ts_tablet_manager.cc:1428] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260806 17:04:09.171402 21778 consensus_queue.cc:260] T 3e7496873b1744a6b43446b2cdaf5940 P eced2e1212f042cfb32e97f349322ed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f4eefe2b9948978263c01fd300bb7c" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42371 } } peers { permanent_uuid: "eced2e1212f042cfb32e97f349322ed0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36253 } } peers { permanent_uuid: "ab3ff57c85b043018039580e8509c746" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43427 } }
I20260806 17:04:09.171759 21475 heartbeater.cc:503] Master 127.9.111.126:35203 was elected leader, sending a full tablet report...
I20260806 17:04:09.172514 21778 ts_tablet_manager.cc:1428] T 3e7496873b1744a6b43446b2cdaf5940 P eced2e1212f042cfb32e97f349322ed0: Time spent starting tablet: real 0.013s user 0.004s sys 0.004s
I20260806 17:04:09.172860 21621 heartbeater.cc:503] Master 127.9.111.126:35203 was elected leader, sending a full tablet report...
I20260806 17:04:09.174033 21782 tablet_bootstrap.cc:492] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746: No bootstrap required, opened a new log
I20260806 17:04:09.174388 21782 ts_tablet_manager.cc:1397] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260806 17:04:09.179252 21782 raft_consensus.cc:348] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f4eefe2b9948978263c01fd300bb7c" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42371 } } peers { permanent_uuid: "eced2e1212f042cfb32e97f349322ed0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36253 } } peers { permanent_uuid: "ab3ff57c85b043018039580e8509c746" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43427 } }
I20260806 17:04:09.179713 21782 raft_consensus.cc:374] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:04:09.179847 21782 raft_consensus.cc:729] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ff57c85b043018039580e8509c746, State: Initialized, Role: FOLLOWER
I20260806 17:04:09.180436 21782 consensus_queue.cc:260] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f4eefe2b9948978263c01fd300bb7c" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42371 } } peers { permanent_uuid: "eced2e1212f042cfb32e97f349322ed0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36253 } } peers { permanent_uuid: "ab3ff57c85b043018039580e8509c746" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43427 } }
I20260806 17:04:09.181188 21758 heartbeater.cc:503] Master 127.9.111.126:35203 was elected leader, sending a full tablet report...
I20260806 17:04:09.181602 21782 ts_tablet_manager.cc:1428] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260806 17:04:09.212546 21476 tablet.cc:2369] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 17:04:09.260447 21759 tablet.cc:2369] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:04:09.297516 21784 raft_consensus.cc:482] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:04:09.297796 21784 raft_consensus.cc:504] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f4eefe2b9948978263c01fd300bb7c" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42371 } } peers { permanent_uuid: "eced2e1212f042cfb32e97f349322ed0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36253 } } peers { permanent_uuid: "ab3ff57c85b043018039580e8509c746" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43427 } }
I20260806 17:04:09.299355 21784 leader_election.cc:288] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eced2e1212f042cfb32e97f349322ed0 (127.9.111.66:36253), ab3ff57c85b043018039580e8509c746 (127.9.111.67:43427)
I20260806 17:04:09.305375 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:09.300137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:55865 (local address 127.9.111.66:36253)
0806 17:04:09.300284 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 17:04:09.300288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:04:09.300569 (+ 281us) server_negotiation.cc:407] Connection header received
0806 17:04:09.300787 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:04:09.300791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:04:09.300843 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:04:09.300890 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:04:09.301353 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.302194 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.303925 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.304580 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.304690 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:09.304750 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 17:04:09.305123 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 17:04:09.305225 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 17:04:09 dist-test-slave-69rg krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035848, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
I20260806 17:04:09.305917 21788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:09.299747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:36253 (local address 127.9.111.65:55865)
0806 17:04:09.300478 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0806 17:04:09.300494 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:04:09.300654 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:04:09.301046 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:04:09.301050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:04:09.301070 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:04:09.301199 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:09.301206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.302405 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.302409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:09.303755 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:09.303769 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.304873 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.304877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:09.304973 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:09.304997 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 17:04:09.305631 (+ 634us) client_negotiation.cc:240] Negotiation successful
0806 17:04:09.305728 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":136,"threads_started":1}
I20260806 17:04:09.306634 21574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7496873b1744a6b43446b2cdaf5940" candidate_uuid: "a6f4eefe2b9948978263c01fd300bb7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eced2e1212f042cfb32e97f349322ed0" is_pre_election: true
I20260806 17:04:09.306993 21574 raft_consensus.cc:2454] T 3e7496873b1744a6b43446b2cdaf5940 P eced2e1212f042cfb32e97f349322ed0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6f4eefe2b9948978263c01fd300bb7c in term 0.
I20260806 17:04:09.307559 21361 leader_election.cc:302] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6f4eefe2b9948978263c01fd300bb7c, eced2e1212f042cfb32e97f349322ed0; no voters:
I20260806 17:04:09.307900 21784 raft_consensus.cc:2781] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:04:09.307976 21784 raft_consensus.cc:482] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:04:09.308014 21784 raft_consensus.cc:3037] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:04:09.309408 21784 raft_consensus.cc:504] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f4eefe2b9948978263c01fd300bb7c" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42371 } } peers { permanent_uuid: "eced2e1212f042cfb32e97f349322ed0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36253 } } peers { permanent_uuid: "ab3ff57c85b043018039580e8509c746" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43427 } }
I20260806 17:04:09.309850 21784 leader_election.cc:288] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c [CANDIDATE]: Term 1 election: Requested vote from peers eced2e1212f042cfb32e97f349322ed0 (127.9.111.66:36253), ab3ff57c85b043018039580e8509c746 (127.9.111.67:43427)
I20260806 17:04:09.310405 21574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7496873b1744a6b43446b2cdaf5940" candidate_uuid: "a6f4eefe2b9948978263c01fd300bb7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eced2e1212f042cfb32e97f349322ed0"
I20260806 17:04:09.310546 21574 raft_consensus.cc:3037] T 3e7496873b1744a6b43446b2cdaf5940 P eced2e1212f042cfb32e97f349322ed0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:04:09.311879 21574 raft_consensus.cc:2454] T 3e7496873b1744a6b43446b2cdaf5940 P eced2e1212f042cfb32e97f349322ed0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f4eefe2b9948978263c01fd300bb7c in term 1.
I20260806 17:04:09.312279 21361 leader_election.cc:302] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6f4eefe2b9948978263c01fd300bb7c, eced2e1212f042cfb32e97f349322ed0; no voters:
I20260806 17:04:09.312559 21784 raft_consensus.cc:2781] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:04:09.312821 21784 raft_consensus.cc:686] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c [term 1 LEADER]: Becoming Leader. State: Replica: a6f4eefe2b9948978263c01fd300bb7c, State: Running, Role: LEADER
I20260806 17:04:09.313220 21784 consensus_queue.cc:237] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f4eefe2b9948978263c01fd300bb7c" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42371 } } peers { permanent_uuid: "eced2e1212f042cfb32e97f349322ed0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36253 } } peers { permanent_uuid: "ab3ff57c85b043018039580e8509c746" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43427 } }
I20260806 17:04:09.316869 21257 catalog_manager.cc:5495] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c reported cstate change: term changed from 0 to 1, leader changed from <none> to a6f4eefe2b9948978263c01fd300bb7c (127.9.111.65). New cstate: current_term: 1 leader_uuid: "a6f4eefe2b9948978263c01fd300bb7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f4eefe2b9948978263c01fd300bb7c" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eced2e1212f042cfb32e97f349322ed0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ff57c85b043018039580e8509c746" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43427 } health_report { overall_health: UNKNOWN } } }
I20260806 17:04:09.325214 21776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:09.300420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:53555 (local address 127.9.111.67:43427)
0806 17:04:09.300571 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 17:04:09.300576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:04:09.317111 (+ 16535us) server_negotiation.cc:407] Connection header received
0806 17:04:09.317262 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:04:09.317265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:04:09.317335 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:04:09.317410 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:04:09.317920 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.318644 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.323583 (+ 4939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.324252 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.324382 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:09.324445 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 17:04:09.324953 (+ 508us) server_negotiation.cc:299] Negotiation successful
0806 17:04:09.325058 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 17:04:09.325654 21789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:09.300298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:43427 (local address 127.9.111.65:53555)
0806 17:04:09.316994 (+ 16696us) negotiation.cc:107] Waiting for socket to connect
0806 17:04:09.317013 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 17:04:09.317152 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:04:09.317545 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:04:09.317552 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:04:09.317575 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:04:09.317713 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:09.317721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.321474 (+ 3753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.321478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:09.323380 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:09.323398 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.324632 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.324636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:09.324760 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:09.324811 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 17:04:09.325403 (+ 592us) client_negotiation.cc:240] Negotiation successful
0806 17:04:09.325498 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16603,"thread_start_us":97,"threads_started":1}
I20260806 17:04:09.326399 21711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7496873b1744a6b43446b2cdaf5940" candidate_uuid: "a6f4eefe2b9948978263c01fd300bb7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ff57c85b043018039580e8509c746" is_pre_election: true
I20260806 17:04:09.326483 21708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7496873b1744a6b43446b2cdaf5940" candidate_uuid: "a6f4eefe2b9948978263c01fd300bb7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ff57c85b043018039580e8509c746"
I20260806 17:04:09.326713 21708 raft_consensus.cc:3037] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:04:09.328243 21708 raft_consensus.cc:2454] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f4eefe2b9948978263c01fd300bb7c in term 1.
W20260806 17:04:09.345911 21622 tablet.cc:2369] T 3e7496873b1744a6b43446b2cdaf5940 P eced2e1212f042cfb32e97f349322ed0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:04:09.376863 21772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:09.372454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40498 (local address 127.9.111.65:42371)
0806 17:04:09.372665 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 17:04:09.372670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:04:09.372688 (+ 18us) server_negotiation.cc:407] Connection header received
0806 17:04:09.372799 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:04:09.372804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:04:09.372893 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:04:09.373026 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:04:09.373517 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.374116 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.375139 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:09.375490 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:09.375617 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:09.375971 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:04:09.376423 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:04:09.376569 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 17:04:09.376631 (+ 62us) server_negotiation.cc:299] Negotiation successful
0806 17:04:09.376691 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260806 17:04:09.384688 21708 raft_consensus.cc:1261] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746 [term 1 FOLLOWER]: Refusing update from remote peer a6f4eefe2b9948978263c01fd300bb7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:04:09.385593 21784 consensus_queue.cc:1035] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ff57c85b043018039580e8509c746" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:04:09.386227 21574 raft_consensus.cc:1261] T 3e7496873b1744a6b43446b2cdaf5940 P eced2e1212f042cfb32e97f349322ed0 [term 1 FOLLOWER]: Refusing update from remote peer a6f4eefe2b9948978263c01fd300bb7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:04:09.386982 21784 consensus_queue.cc:1035] T 3e7496873b1744a6b43446b2cdaf5940 P a6f4eefe2b9948978263c01fd300bb7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eced2e1212f042cfb32e97f349322ed0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:04:09.397680 21794 mvcc.cc:205] Tried to move back new op lower bound from 7315602839069794304 to 7315602838790721536. Current Snapshot: MvccSnapshot[applied={T|T < 7315602839069794304}]
I20260806 17:04:09.404860 21795 mvcc.cc:205] Tried to move back new op lower bound from 7315602839069794304 to 7315602838790721536. Current Snapshot: MvccSnapshot[applied={T|T < 7315602839069794304}]
I20260806 17:04:09.414953 21796 mvcc.cc:205] Tried to move back new op lower bound from 7315602839069794304 to 7315602838790721536. Current Snapshot: MvccSnapshot[applied={T|T < 7315602839069794304}]
I20260806 17:04:09.423205 21256 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:04:09.423426 21256 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:04:09.426913 21256 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51b5baf21ef6430380ad8e2acfb8d221: Sending DeleteTablet for 3 replicas of tablet 3e7496873b1744a6b43446b2cdaf5940
I20260806 17:04:09.427934 21409 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e7496873b1744a6b43446b2cdaf5940 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:04:09 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:52790
I20260806 17:04:09.428944 21691 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e7496873b1744a6b43446b2cdaf5940 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:04:09 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:40662
I20260806 17:04:09.429916 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 21342
I20260806 17:04:09.428380 21532 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e7496873b1744a6b43446b2cdaf5940 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:04:09 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:33170
I20260806 17:04:09.439402 21804 tablet_replica.cc:330] T 3e7496873b1744a6b43446b2cdaf5940 P eced2e1212f042cfb32e97f349322ed0: stopping tablet replica
I20260806 17:04:09.439908 21804 raft_consensus.cc:2229] T 3e7496873b1744a6b43446b2cdaf5940 P eced2e1212f042cfb32e97f349322ed0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:04:09.440241 21804 raft_consensus.cc:2258] T 3e7496873b1744a6b43446b2cdaf5940 P eced2e1212f042cfb32e97f349322ed0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:04:09.443414 21803 tablet_replica.cc:330] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746: stopping tablet replica
W20260806 17:04:09.443605 21241 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.111.65:42371 (error 108)
I20260806 17:04:09.443790 21803 raft_consensus.cc:2229] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:04:09.444144 21803 raft_consensus.cc:2258] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:04:09.444697 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 21483
I20260806 17:04:09.447865 21803 ts_tablet_manager.cc:1905] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:04:09.448575 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:09.448272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:42371 (local address 127.0.0.1:40514)
0806 17:04:09.448390 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 17:04:09.448473 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.65:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260806 17:04:09.449090 21241 catalog_manager.cc:4584] TS a6f4eefe2b9948978263c01fd300bb7c (127.9.111.65:42371): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e7496873b1744a6b43446b2cdaf5940: Network error: Client connection negotiation failed: client connection to 127.9.111.65:42371: connect: Connection refused (error 111)
I20260806 17:04:09.450935 21803 ts_tablet_manager.cc:1918] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:04:09.451049 21803 log.cc:1196] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/3e7496873b1744a6b43446b2cdaf5940
I20260806 17:04:09.451391 21803 ts_tablet_manager.cc:1939] T 3e7496873b1744a6b43446b2cdaf5940 P ab3ff57c85b043018039580e8509c746: Deleting consensus metadata
I20260806 17:04:09.452895 21243 catalog_manager.cc:4841] TS ab3ff57c85b043018039580e8509c746 (127.9.111.67:43427): tablet 3e7496873b1744a6b43446b2cdaf5940 (table test-table [id=89ad3013be9543a8b606977f74cf781e]) successfully deleted
W20260806 17:04:09.457103 21771 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:04:09.456376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:36253 (local address 127.0.0.1:48508)
0806 17:04:09.456581 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 17:04:09.456599 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 17:04:09.456715 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:04:09.456975 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.66:36253: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":112}
I20260806 17:04:09.457800 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 21626
I20260806 17:04:09.468586 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 21222
I20260806 17:04:09.476027 9661 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 17:04:09.822154 9661 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 17:04:09.822363 9661 mini_ranger.cc:66] Stopping Ranger...
I20260806 17:04:10.168840 9661 mini_ranger.cc:68] Stopped Ranger
2026-08-06 17:04:10.169 UTC [20785] LOG: received smart shutdown request
2026-08-06 17:04:10.173 UTC [20785] LOG: background worker "logical replication launcher" (PID 20794) exited with exit code 1
2026-08-06 17:04:10.173 UTC [20789] LOG: shutting down
2026-08-06 17:04:10.415 UTC [20785] LOG: database system is shut down
2026-08-06T17:04:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50508 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:04:10 dist-test-slave-69rg krb5kdc[21814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:04:10 dist-test-slave-69rg krb5kdc[21814](info): set up 2 sockets
Aug 06 17:04:10 dist-test-slave-69rg krb5kdc[21814](info): commencing operation
krb5kdc: starting...
W20260806 17:04:11.896502 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.321s user 0.004s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:04:12 dist-test-slave-69rg krb5kdc[21814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:04:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:04:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.111.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.111.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.111.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/rangeradmin_127.9.111.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.111.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/rangeradmin_127.9.111.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.111.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.111.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.111.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/rangerlookup_127.9.111.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.111.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/rangerlookup_127.9.111.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.111.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.111.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.111.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/HTTP_127.9.111.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.111.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/HTTP_127.9.111.84@KRBTEST.COM.keytab.
I20260806 17:04:12.358994 9661 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7cxihq/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/postgres -l logfile start
2026-08-06 17:04:16.207 UTC [21845] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 17:04:16.207 UTC [21845] LOG: listening on IPv4 address "127.9.111.84", port 60039
2026-08-06 17:04:16.212 UTC [21845] LOG: listening on Unix socket "/tmp/.s.PGSQL.60039"
2026-08-06 17:04:16.236 UTC [21848] LOG: database system was shut down at 2026-08-06 17:04:14 UTC
2026-08-06 17:04:16.240 UTC [21845] LOG: database system is ready to accept connections
I20260806 17:04:17.208999 9661 mini_postgres.cc:96] Postgres bound to 60039
2026-08-06 17:04:17.216 UTC [21857] FATAL: database "slave" does not exist
127.9.111.84:60039 - accepting connections
I20260806 17:04:17.217555 9661 mini_ranger.cc:161] Starting Ranger...
I20260806 17:04:17.235623 9661 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 17:04:17.779250 9661 mini_ranger.cc:87] Created ranger Postgres database
I20260806 17:04:17.779342 9661 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ranger-admin
2026-08-06 17:04:18,063 [I] DB FLAVOR :POSTGRES
2026-08-06 17:04:18,064 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 17:04:18,064 [I] Checking connection..
2026-08-06 17:04:18,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 17:04:18,330 [I] Checking connection passed.
2026-08-06 17:04:18,330 [I] --------- Verifying version history table ---------
2026-08-06 17:04:18,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 17:04:18,590 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 17:04:18,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 17:04:18,839 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 17:04:18,839 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 17:04:18,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 17:04:19.087 UTC [21925] WARNING: there is no transaction in progress
2026-08-06 17:04:19,095 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 17:04:19,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 17:04:19,370 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 17:04:19,370 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 17:04:19,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 17:04:19,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 17:04:19,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-69rg.c.gcp-upstream.internal','N') ;"
2026-08-06 17:04:20,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 17:04:20,372 [I] Table x_portal_user does not exist in database ranger
2026-08-06 17:04:20,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 17:04:20,618 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 17:04:20,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 17:04:20,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 17:04:21,099 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 17:04:21,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 17:04:22.160 UTC [22038] WARNING: there is no transaction in progress
2026-08-06 17:04:22.182 UTC [22038] WARNING: there is no transaction in progress
2026-08-06 17:04:22.204 UTC [22038] WARNING: there is no transaction in progress
2026-08-06 17:04:22.224 UTC [22038] WARNING: there is no transaction in progress
2026-08-06 17:04:22.245 UTC [22038] WARNING: there is no transaction in progress
2026-08-06 17:04:22.270 UTC [22038] WARNING: there is no transaction in progress
2026-08-06 17:04:22.343 UTC [22038] WARNING: there is no transaction in progress
2026-08-06 17:04:22.351 UTC [22038] WARNING: there is no transaction in progress
2026-08-06 17:04:22.368 UTC [22038] WARNING: there is no transaction in progress
2026-08-06 17:04:22.379 UTC [22038] WARNING: there is no transaction in progress
2026-08-06 17:04:22.391 UTC [22038] WARNING: there is no transaction in progress
2026-08-06 17:04:22.408 UTC [22038] WARNING: there is no transaction in progress
2026-08-06 17:04:22.420 UTC [22038] WARNING: there is no transaction in progress
2026-08-06 17:04:22.428 UTC [22038] WARNING: there is no transaction in progress
2026-08-06 17:04:22.437 UTC [22038] WARNING: there is no transaction in progress
2026-08-06 17:04:22.447 UTC [22038] WARNING: there is no transaction in progress
2026-08-06 17:04:23,261 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 17:04:23,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 17:04:23,516 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 17:04:23,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 17:04:23,790 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 17:04:23,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 17:04:24,050 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 17:04:24,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 17:04:24,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 17:04:24,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-69rg.c.gcp-upstream.internal';"
2026-08-06 17:04:24,770 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 17:04:24,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 17:04:25,054 [I] DB_PATCHES have already been applied
I20260806 17:04:25.060945 9661 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ranger-admin:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7cxihq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 17:04:25.061017 9661 mini_ranger.cc:193] Using host: 127.9.111.84
I20260806 17:04:25.066220 9661 mini_ranger.cc:227] Ranger admin URL: http://127.9.111.84:49577
Aug 06, 2026 5:04:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 5:04:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 5:04:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 5:04:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 5:04:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 5:04:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 5:04:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 17:04:26.200861 9661 mini_ranger.cc:160] Time spent starting Ranger: real 8.983s user 0.002s sys 0.012s
Aug 06, 2026 5:04:47 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 5:04:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 5:04:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 5:04:47 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 5:04:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 5:04:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 5:04:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 5:04:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 5:04:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 5:04:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 5:04:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 5:04:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 5:04:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 5:04:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 5:04:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 5:04:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 5:04:47 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 5:04:48 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 17:04:49.307420 9661 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.111.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.111.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.111.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/rangerkms_127.9.111.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.111.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/rangerkms_127.9.111.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.111.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.111.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.111.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/HTTP_127.9.111.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.111.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/HTTP_127.9.111.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 17:04:49 dist-test-slave-69rg krb5kdc[21814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035889, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 17:04:49.403771 9661 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 17:04:49.414906 9661 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 17:04:50.426273 9661 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 17:04:50.426373 9661 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ranger-kms
I20260806 17:04:50.426393 9661 mini_ranger_kms.cc:227] Using postgres at 127.9.111.84:60039
2026-08-06 17:04:50,650 [I] DB FLAVOR :POSTGRES
2026-08-06 17:04:50,651 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 17:04:50,651 [I] Checking connection
2026-08-06 17:04:50,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 17:04:50,903 [I] connection success
2026-08-06 17:04:50,904 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 17:04:50,904 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 17:04:50,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 17:04:51,156 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 17:04:51,156 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 17:04:51,157 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 17:04:51,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.111.84:60039/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 17:04:51,418 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 17:04:51.909577 9661 mini_ranger_kms.cc:310] Created kms service
I20260806 17:04:52.155200 9661 mini_ranger_kms.cc:326] Created kudu user
I20260806 17:04:52.201072 9661 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 17:04:52 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035889, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
I20260806 17:04:52.617149 9661 mini_ranger_kms.cc:384] Added ranger policy
I20260806 17:04:52.617354 9661 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ranger-kms:/tmp/dist-test-task7cxihq/build/debug/bin/postgresql.jar:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7cxihq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 17:04:52.617383 9661 mini_ranger_kms.cc:240] Using host: 127.9.111.84
I20260806 17:04:52.621865 9661 mini_ranger_kms.cc:276] Ranger KMS PID: 22225
I20260806 17:04:52.621958 9661 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.111.84:48429
Aug 06, 2026 5:04:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 5:04:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 5:04:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 5:04:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 5:04:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 5:04:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 17:04:54 dist-test-slave-69rg krb5kdc[21814](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786035894, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.111.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:04:54.958112 9661 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.554s user 0.009s sys 0.200s
I20260806 17:04:54.958252 9661 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 17:04:54.958338 9661 mini_ranger_kms.cc:401] 127.9.111.84:48429/kms/v1/keys
Aug 06 17:04:55 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035894, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.111.84@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
Aug 06 17:04:55 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035894, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.111.84@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
Aug 06 17:04:58 dist-test-slave-69rg krb5kdc[21814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:04:58.089946 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:43801
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:43801
--ranger_config_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.111.84:48429/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:04:58.229038 22276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:04:58.229326 22276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:04:58.229389 22276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:04:58.233572 22276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:04:58.233654 22276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:04:58.233673 22276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:04:58.233695 22276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:04:58.233738 22276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:04:58.239625 22276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.111.84:48429/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:43801
--ranger_config_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:43801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:04:58.240765 22276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:04:58.241883 22276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:04:58.250967 22282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:04:58.251075 22281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:04:58.250974 22284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:04:58.251314 22276 server_base.cc:1034] running on GCE node
I20260806 17:04:58.252019 22276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:04:58.253497 22276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:04:58.254719 22276 hybrid_clock.cc:648] HybridClock initialized: now 1786035898254707 us; error 32 us; skew 500 ppm
Aug 06 17:04:58 dist-test-slave-69rg krb5kdc[21814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035898, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:04:58.262351 22276 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:04:58.263387 22276 webserver.cc:460] Webserver started at http://127.9.111.126:40257/ using document root <none> and password file <none>
I20260806 17:04:58.263963 22276 fs_manager.cc:359] Metadata directory not provided
I20260806 17:04:58.264020 22276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:04:58.264254 22276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 17:04:58 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035898, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
I20260806 17:04:58.310685 22276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "203b4541d648435b97f82aa04d720ecb"
format_stamp: "Formatted at 2026-08-06 17:04:58 on dist-test-slave-69rg"
server_key: "d2ab79b106d92d60d9da8f7774945ac0"
server_key_iv: "7e68ef66a29c635472908b311d0c2f16"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:58.311358 22276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "203b4541d648435b97f82aa04d720ecb"
format_stamp: "Formatted at 2026-08-06 17:04:58 on dist-test-slave-69rg"
server_key: "d2ab79b106d92d60d9da8f7774945ac0"
server_key_iv: "7e68ef66a29c635472908b311d0c2f16"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:58.316064 22276 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 17:04:58.333788 22292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:04:58.335284 22276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 17:04:58.335439 22276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "203b4541d648435b97f82aa04d720ecb"
format_stamp: "Formatted at 2026-08-06 17:04:58 on dist-test-slave-69rg"
server_key: "d2ab79b106d92d60d9da8f7774945ac0"
server_key_iv: "7e68ef66a29c635472908b311d0c2f16"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:58.335579 22276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:04:58.358713 22276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:04:58.361796 22276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:04:58.362071 22276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:04:58.370749 22276 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:43801
I20260806 17:04:58.370766 22346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:43801 every 8 connection(s)
I20260806 17:04:58.371748 22276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:04:58.375226 22347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:04:58.380280 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 22276
I20260806 17:04:58.380456 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:04:58.380957 9661 external_mini_cluster.cc:1428] Setting key f881539b2cf3074af3f0a55d5ebe70ea
I20260806 17:04:58.382537 22347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb: Bootstrap starting.
I20260806 17:04:58.385361 22347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb: Neither blocks nor log segments found. Creating new log.
I20260806 17:04:58.386349 22347 log.cc:824] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb: Log is configured to *not* fsync() on all Append() calls
I20260806 17:04:58.388669 22347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb: No bootstrap required, opened a new log
Aug 06 17:04:58 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:04:58.393337 22347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203b4541d648435b97f82aa04d720ecb" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43801 } }
I20260806 17:04:58.393692 22347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:04:58.393765 22347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 203b4541d648435b97f82aa04d720ecb, State: Initialized, Role: FOLLOWER
I20260806 17:04:58.394272 22347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203b4541d648435b97f82aa04d720ecb" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43801 } }
I20260806 17:04:58.394428 22347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:04:58.394513 22347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:04:58.394608 22347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:04:58.395666 22347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203b4541d648435b97f82aa04d720ecb" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43801 } }
I20260806 17:04:58.396024 22347 leader_election.cc:302] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 203b4541d648435b97f82aa04d720ecb; no voters:
I20260806 17:04:58.396353 22347 leader_election.cc:288] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:04:58.397228 22352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:04:58.397517 22352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [term 1 LEADER]: Becoming Leader. State: Replica: 203b4541d648435b97f82aa04d720ecb, State: Running, Role: LEADER
I20260806 17:04:58.397864 22347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:04:58.397893 22352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203b4541d648435b97f82aa04d720ecb" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43801 } }
I20260806 17:04:58.398499 22350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:58.382950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38882 (local address 127.9.111.126:43801)
0806 17:04:58.383489 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0806 17:04:58.383507 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 17:04:58.383544 (+ 37us) server_negotiation.cc:407] Connection header received
0806 17:04:58.384317 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:04:58.384336 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:04:58.384690 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:04:58.385076 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:04:58.386275 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:58.387682 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:58.388718 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:58.389292 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:58.389475 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:58.392538 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:04:58.392545 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:04:58.392560 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:04:58.392588 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:04:58.394745 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:04:58.395224 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:04:58.395230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:04:58.395236 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:04:58.395357 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:04:58.395690 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:04:58.395693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:04:58.395695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:04:58.396230 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:04:58.396456 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 17:04:58.396781 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 17:04:58.397115 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":211,"threads_started":1}
I20260806 17:04:58.399127 22352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 203b4541d648435b97f82aa04d720ecb. Latest consensus state: current_term: 1 leader_uuid: "203b4541d648435b97f82aa04d720ecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203b4541d648435b97f82aa04d720ecb" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43801 } } }
I20260806 17:04:58.399250 22352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [sys.catalog]: This master's current role is: LEADER
I20260806 17:04:58.399631 22361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:04:58.400547 22354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "203b4541d648435b97f82aa04d720ecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203b4541d648435b97f82aa04d720ecb" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43801 } } }
I20260806 17:04:58.400633 22354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [sys.catalog]: This master's current role is: LEADER
I20260806 17:04:58.402930 22361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:04:58.403594 22347 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 17:04:58.409694 22361 catalog_manager.cc:1347] Generated new cluster ID: 3c0b22b40ede45889c99446dee19bf97
I20260806 17:04:58.409761 22361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:04:58.447659 22361 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:04:58.449074 22361 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:04:58.457954 22361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb: Generated new TSK 0
I20260806 17:04:58.458889 22361 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
I20260806 17:04:59.177156 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.111.84:48429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:43801
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:04:59.300871 22392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:04:59.301144 22392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:04:59.301188 22392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:04:59.305572 22392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:04:59.305639 22392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:04:59.305698 22392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:04:59.311770 22392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.111.84:48429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:43801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:04:59.313115 22392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:04:59.314625 22392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:04:59.327159 22400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:04:59.327225 22399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:04:59.327167 22402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:04:59.327648 22392 server_base.cc:1034] running on GCE node
I20260806 17:04:59.328106 22392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:04:59.328819 22392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:04:59.330011 22392 hybrid_clock.cc:648] HybridClock initialized: now 1786035899329986 us; error 37 us; skew 500 ppm
Aug 06 17:04:59 dist-test-slave-69rg krb5kdc[21814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035899, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:04:59.334070 22392 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:04:59.335222 22392 webserver.cc:460] Webserver started at http://127.9.111.65:36693/ using document root <none> and password file <none>
I20260806 17:04:59.335937 22392 fs_manager.cc:359] Metadata directory not provided
I20260806 17:04:59.336033 22392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:04:59.336279 22392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 17:04:59 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035899, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
I20260806 17:04:59.353894 22392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "1a907f91e7f74fa2b8542c44441c8fe7"
format_stamp: "Formatted at 2026-08-06 17:04:59 on dist-test-slave-69rg"
server_key: "20e1e672318ef581a3f347eacacdd744"
server_key_iv: "cd97598eec694006662287524182aa7a"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:59.354921 22392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "1a907f91e7f74fa2b8542c44441c8fe7"
format_stamp: "Formatted at 2026-08-06 17:04:59 on dist-test-slave-69rg"
server_key: "20e1e672318ef581a3f347eacacdd744"
server_key_iv: "cd97598eec694006662287524182aa7a"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:59.363627 22392 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260806 17:04:59.382196 22409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:04:59.383968 22392 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260806 17:04:59.384186 22392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "1a907f91e7f74fa2b8542c44441c8fe7"
format_stamp: "Formatted at 2026-08-06 17:04:59 on dist-test-slave-69rg"
server_key: "20e1e672318ef581a3f347eacacdd744"
server_key_iv: "cd97598eec694006662287524182aa7a"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:59.384405 22392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:04:59.414549 22392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:04:59.418565 22392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:04:59.418855 22392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:04:59.419710 22392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:04:59.420933 22392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:04:59.421044 22392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:04:59.421175 22392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:04:59.421257 22392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:04:59.435333 22392 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:32795
I20260806 17:04:59.436796 22392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:04:59.446074 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 22392
I20260806 17:04:59.446096 22527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:32795 every 8 connection(s)
I20260806 17:04:59.446226 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:04:59.446650 9661 external_mini_cluster.cc:1428] Setting key 0acbcc581ba4dfab89d96dc0e0e7fd6e
WARNING: no policy specified for kudu/127.9.111.66@KRBTEST.COM; defaulting to no policy
Aug 06 17:04:59 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035899, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:04:59.494167 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:59.448631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:43801 (local address 127.9.111.65:56963)
0806 17:04:59.458262 (+ 9631us) negotiation.cc:107] Waiting for socket to connect
0806 17:04:59.458349 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0806 17:04:59.459513 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:04:59.460572 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:04:59.460582 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:04:59.460999 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:04:59.461741 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:59.461763 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:59.463034 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:59.463038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:59.464467 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:59.464479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:59.480890 (+ 16411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:59.480896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:59.481195 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:59.482084 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:04:59.482109 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:04:59.487245 (+ 5136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:04:59.490057 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:04:59.490067 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:04:59.490089 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:04:59.490582 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:04:59.490968 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:04:59.490972 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:04:59.490974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:04:59.491186 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:04:59.491790 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:04:59.491797 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:04:59.492288 (+ 491us) client_negotiation.cc:769] Sending connection context
0806 17:04:59.492575 (+ 287us) client_negotiation.cc:240] Negotiation successful
0806 17:04:59.492971 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9282,"thread_start_us":177,"threads_started":1}
I20260806 17:04:59.495002 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:59.450389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:56963 (local address 127.9.111.126:43801)
0806 17:04:59.460015 (+ 9626us) server_negotiation.cc:206] Beginning negotiation
0806 17:04:59.460025 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 17:04:59.460040 (+ 15us) server_negotiation.cc:407] Connection header received
0806 17:04:59.460084 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:04:59.460087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:04:59.460135 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:04:59.460224 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:04:59.461964 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:59.462864 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:59.471403 (+ 8539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:59.471814 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:59.471956 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:59.487628 (+ 15672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:04:59.487633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:04:59.487637 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:04:59.487674 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:04:59.489860 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:04:59.490732 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:04:59.490736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:04:59.490738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:04:59.490843 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:04:59.491331 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:04:59.491334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:04:59.491336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:04:59.491617 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:04:59.491728 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 17:04:59.494675 (+ 2947us) server_negotiation.cc:299] Negotiation successful
0806 17:04:59.494821 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9537,"thread_start_us":118,"threads_started":1}
I20260806 17:04:59.497109 22528 heartbeater.cc:347] Connected to a master server at 127.9.111.126:43801
I20260806 17:04:59.497406 22528 heartbeater.cc:464] Registering TS with master...
I20260806 17:04:59.498121 22528 heartbeater.cc:511] Master 127.9.111.126:43801 requested a full tablet report, sending...
I20260806 17:04:59.500057 22309 ts_manager.cc:194] Registered new tserver with Master: 1a907f91e7f74fa2b8542c44441c8fe7 (127.9.111.65:32795)
I20260806 17:04:59.501690 22309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:56963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
I20260806 17:04:59.527906 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.111.84:48429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:43801
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 17:04:59 dist-test-slave-69rg krb5kdc[21814](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786035899, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 17:04:59.692763 22537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:04:59.693182 22537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:04:59.693269 22537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:04:59.701241 22537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:04:59.701344 22537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:04:59.701442 22537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:04:59.710479 22537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.111.84:48429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:43801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:04:59.712050 22537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:04:59.713716 22537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:04:59.728648 22544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:04:59.728920 22546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:04:59.729504 22537 server_base.cc:1034] running on GCE node
W20260806 17:04:59.729283 22543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:04:59.730283 22537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:04:59.731178 22537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:04:59.732375 22537 hybrid_clock.cc:648] HybridClock initialized: now 1786035899732301 us; error 90 us; skew 500 ppm
Aug 06 17:04:59 dist-test-slave-69rg krb5kdc[21814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035899, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:04:59.736402 22537 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:04:59.737617 22537 webserver.cc:460] Webserver started at http://127.9.111.66:33531/ using document root <none> and password file <none>
I20260806 17:04:59.738461 22537 fs_manager.cc:359] Metadata directory not provided
I20260806 17:04:59.738559 22537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:04:59.738811 22537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 17:04:59 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035899, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
I20260806 17:04:59.765657 22537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "61742a45ecb0458aa19845e93cda241b"
format_stamp: "Formatted at 2026-08-06 17:04:59 on dist-test-slave-69rg"
server_key: "a32fc4f5b94ebdb73f9531cd8703bf9a"
server_key_iv: "232da25682469e5693fab80b8d869c0b"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:59.768013 22537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "61742a45ecb0458aa19845e93cda241b"
format_stamp: "Formatted at 2026-08-06 17:04:59 on dist-test-slave-69rg"
server_key: "a32fc4f5b94ebdb73f9531cd8703bf9a"
server_key_iv: "232da25682469e5693fab80b8d869c0b"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:59.774530 22537 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 17:04:59.801926 22553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:04:59.803900 22537 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260806 17:04:59.804070 22537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "61742a45ecb0458aa19845e93cda241b"
format_stamp: "Formatted at 2026-08-06 17:04:59 on dist-test-slave-69rg"
server_key: "a32fc4f5b94ebdb73f9531cd8703bf9a"
server_key_iv: "232da25682469e5693fab80b8d869c0b"
server_key_version: "kuduclusterkey@0"
I20260806 17:04:59.804253 22537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:04:59.842031 22537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:04:59.846118 22537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:04:59.846343 22537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:04:59.847254 22537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:04:59.848608 22537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:04:59.848685 22537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:04:59.848757 22537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:04:59.848802 22537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:04:59.864969 22537 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:35461
I20260806 17:04:59.866143 22537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:04:59.874142 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 22537
I20260806 17:04:59.874295 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:04:59.874608 9661 external_mini_cluster.cc:1428] Setting key 8905eedf9364979d15bf1be7ad2995b0
I20260806 17:04:59.878037 22671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:35461 every 8 connection(s)
Aug 06 17:04:59 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035899, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:04:59.898981 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:59.880749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:55575 (local address 127.9.111.126:43801)
0806 17:04:59.880906 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 17:04:59.880912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:04:59.882121 (+ 1209us) server_negotiation.cc:407] Connection header received
0806 17:04:59.883394 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:04:59.883398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:04:59.883455 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:04:59.883511 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:04:59.885181 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:59.886077 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:59.887810 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:04:59.888151 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:04:59.888237 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:59.894159 (+ 5922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:04:59.894164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:04:59.894167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:04:59.894201 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:04:59.896144 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:04:59.896858 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:04:59.896862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:04:59.896864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:04:59.896958 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:04:59.897388 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:04:59.897392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:04:59.897394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:04:59.897637 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:04:59.897707 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 17:04:59.898637 (+ 930us) server_negotiation.cc:299] Negotiation successful
0806 17:04:59.898803 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
WARNING: no policy specified for kudu/127.9.111.67@KRBTEST.COM; defaulting to no policy
I20260806 17:04:59.900862 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:04:59.880745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:43801 (local address 127.9.111.66:55575)
0806 17:04:59.881964 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0806 17:04:59.882021 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 17:04:59.883106 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:04:59.883945 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:04:59.883954 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:04:59.884327 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:04:59.885000 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:59.885020 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:59.886246 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:59.886249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:59.887650 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:04:59.887661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:04:59.888460 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:04:59.888464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:04:59.888666 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:04:59.889563 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:04:59.889585 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:04:59.892874 (+ 3289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:04:59.896307 (+ 3433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:04:59.896314 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:04:59.896335 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:04:59.896714 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:04:59.897067 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:04:59.897070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:04:59.897072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:04:59.897278 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:04:59.897847 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:04:59.897853 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:04:59.898237 (+ 384us) client_negotiation.cc:769] Sending connection context
0806 17:04:59.899230 (+ 993us) client_negotiation.cc:240] Negotiation successful
0806 17:04:59.899659 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"thread_start_us":146,"threads_started":1}
I20260806 17:04:59.902314 22672 heartbeater.cc:347] Connected to a master server at 127.9.111.126:43801
I20260806 17:04:59.902583 22672 heartbeater.cc:464] Registering TS with master...
I20260806 17:04:59.903198 22672 heartbeater.cc:511] Master 127.9.111.126:43801 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.67@KRBTEST.COM" created.
I20260806 17:04:59.904803 22309 ts_manager.cc:194] Registered new tserver with Master: 61742a45ecb0458aa19845e93cda241b (127.9.111.66:35461)
I20260806 17:04:59.905572 22309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:55575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
I20260806 17:04:59.929309 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.111.84:48429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:43801
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:05:00.088917 22677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:00.089399 22677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:00.089535 22677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:00.096094 22677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:00.096290 22677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:00.096448 22677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:05:00.106176 22677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.111.84:48429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:43801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:00.107973 22677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:00.109453 22677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Aug 06 17:05:00 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035899, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
W20260806 17:05:00.126142 22682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:00.130862 22683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:00.134472 22685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:00.138957 22677 server_base.cc:1034] running on GCE node
I20260806 17:05:00.139775 22677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:00.140748 22677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:00.148173 22677 hybrid_clock.cc:648] HybridClock initialized: now 1786035900148077 us; error 101 us; skew 500 ppm
Aug 06 17:05:00 dist-test-slave-69rg krb5kdc[21814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035900, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:00.152180 22677 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:05:00.153721 22677 webserver.cc:460] Webserver started at http://127.9.111.67:35895/ using document root <none> and password file <none>
I20260806 17:05:00.154567 22677 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:00.154666 22677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:00.154937 22677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 17:05:00 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035900, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
I20260806 17:05:00.186054 22677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "11ce4c3e93df4518b231453bfe9f3577"
format_stamp: "Formatted at 2026-08-06 17:05:00 on dist-test-slave-69rg"
server_key: "aff7b7c4a8ed601b8a765f2f2700e1c8"
server_key_iv: "868e0ebdbb0f96d7fdc20224bb758f70"
server_key_version: "kuduclusterkey@0"
I20260806 17:05:00.187050 22677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "11ce4c3e93df4518b231453bfe9f3577"
format_stamp: "Formatted at 2026-08-06 17:05:00 on dist-test-slave-69rg"
server_key: "aff7b7c4a8ed601b8a765f2f2700e1c8"
server_key_iv: "868e0ebdbb0f96d7fdc20224bb758f70"
server_key_version: "kuduclusterkey@0"
I20260806 17:05:00.194053 22677 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.001s
I20260806 17:05:00.218340 22692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:00.220281 22677 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260806 17:05:00.220443 22677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "11ce4c3e93df4518b231453bfe9f3577"
format_stamp: "Formatted at 2026-08-06 17:05:00 on dist-test-slave-69rg"
server_key: "aff7b7c4a8ed601b8a765f2f2700e1c8"
server_key_iv: "868e0ebdbb0f96d7fdc20224bb758f70"
server_key_version: "kuduclusterkey@0"
I20260806 17:05:00.220599 22677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:00.248773 22677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:00.252832 22677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:00.253057 22677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:00.253850 22677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:05:00.254964 22677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:05:00.255033 22677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:00.255088 22677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:05:00.255120 22677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:00.267361 22677 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:41261
I20260806 17:05:00.267524 22808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:41261 every 8 connection(s)
I20260806 17:05:00.268625 22677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:05:00.269719 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 22677
I20260806 17:05:00.269896 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:05:00.270346 9661 external_mini_cluster.cc:1428] Setting key 85dd9dee82c74a31a05c75050d2acbe2
Aug 06 17:05:00 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035900, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:05:00.288904 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:00.274671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:42043 (local address 127.9.111.126:43801)
0806 17:05:00.274794 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:00.274798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:00.274816 (+ 18us) server_negotiation.cc:407] Connection header received
0806 17:05:00.275932 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:00.275936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:00.275995 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:00.276091 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:00.277665 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:00.278547 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:00.280355 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:00.280701 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:00.280828 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:00.284122 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:00.284126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:00.284129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:00.284160 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:00.286111 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:00.286829 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:00.286833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:00.286835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:00.286982 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:00.287432 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:00.287436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:00.287438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:00.287679 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:00.287768 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:00.288562 (+ 794us) server_negotiation.cc:299] Negotiation successful
0806 17:05:00.288725 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 17:05:00.291028 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:00.273499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:43801 (local address 127.9.111.67:42043)
0806 17:05:00.274247 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:00.274311 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:00.275617 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:00.276286 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:00.276295 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:00.276721 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:00.277484 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:00.277504 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:00.278697 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:00.278701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:00.280205 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:00.280218 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:00.280881 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:00.280885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:00.281104 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:00.281894 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:00.281917 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:00.283910 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:00.286260 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:00.286268 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:00.286287 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:00.286683 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:00.287108 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:00.287111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:00.287114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:00.287314 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:00.287859 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:00.287866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:00.288241 (+ 375us) client_negotiation.cc:769] Sending connection context
0806 17:05:00.289377 (+ 1136us) client_negotiation.cc:240] Negotiation successful
0806 17:05:00.289756 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":186,"threads_started":1}
I20260806 17:05:00.292579 22809 heartbeater.cc:347] Connected to a master server at 127.9.111.126:43801
I20260806 17:05:00.292910 22809 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:00.293592 22809 heartbeater.cc:511] Master 127.9.111.126:43801 requested a full tablet report, sending...
I20260806 17:05:00.294947 22309 ts_manager.cc:194] Registered new tserver with Master: 11ce4c3e93df4518b231453bfe9f3577 (127.9.111.67:41261)
I20260806 17:05:00.295787 22309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:42043
I20260806 17:05:00.296819 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 17:05:00.298022 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 22392
I20260806 17:05:00.308079 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 22537
I20260806 17:05:00.316159 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 22677
I20260806 17:05:00.328963 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 22276
I20260806 17:05:00.337967 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:43801
--webserver_interface=127.9.111.126
--webserver_port=40257
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:43801
--ranger_config_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.111.84:48429/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.111.84:48429/kms with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:05:00.464337 22812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:00.464668 22812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:00.464723 22812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:00.469379 22812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:05:00.469475 22812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:00.469496 22812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:00.469512 22812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:05:00.469528 22812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:05:00.475344 22812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.111.84:48429/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:43801
--ranger_config_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:43801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=40257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:00.476462 22812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:00.477470 22812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:00.486703 22817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:00.486706 22820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:00.486726 22818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:00.486770 22812 server_base.cc:1034] running on GCE node
I20260806 17:05:00.487622 22812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:00.488750 22812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:00.489894 22812 hybrid_clock.cc:648] HybridClock initialized: now 1786035900489861 us; error 43 us; skew 500 ppm
Aug 06 17:05:00 dist-test-slave-69rg krb5kdc[21814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035900, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:00.492725 22812 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:05:00.493678 22812 webserver.cc:460] Webserver started at http://127.9.111.126:40257/ using document root <none> and password file <none>
I20260806 17:05:00.494311 22812 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:00.494414 22812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 17:05:00 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035900, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
I20260806 17:05:00.545419 22812 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 17:05:00.548048 22828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:00.549432 22812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 17:05:00.549590 22812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "203b4541d648435b97f82aa04d720ecb"
format_stamp: "Formatted at 2026-08-06 17:04:58 on dist-test-slave-69rg"
server_key: "d2ab79b106d92d60d9da8f7774945ac0"
server_key_iv: "7e68ef66a29c635472908b311d0c2f16"
server_key_version: "kuduclusterkey@0"
I20260806 17:05:00.550241 22812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:00.568605 22812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:00.571250 22812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:00.571470 22812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:00.579281 22812 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:43801
I20260806 17:05:00.579288 22883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:43801 every 8 connection(s)
I20260806 17:05:00.580413 22812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:05:00.584728 22884 sys_catalog.cc:263] Verifying existing consensus state
I20260806 17:05:00.586416 22884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb: Bootstrap starting.
I20260806 17:05:00.587131 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 22812
I20260806 17:05:00.587590 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.111.84:48429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:32795
--local_ip_for_outbound_sockets=127.9.111.65
--tserver_master_addrs=127.9.111.126:43801
--webserver_port=36693
--webserver_interface=127.9.111.65
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.111.84:48429/kms with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:05:00.603348 22884 log.cc:824] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:00.610234 22884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 17:05:00.610669 22884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb: Bootstrap complete.
I20260806 17:05:00.614970 22884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203b4541d648435b97f82aa04d720ecb" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43801 } }
I20260806 17:05:00.615396 22884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 203b4541d648435b97f82aa04d720ecb, State: Initialized, Role: FOLLOWER
I20260806 17:05:00.615942 22884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203b4541d648435b97f82aa04d720ecb" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43801 } }
I20260806 17:05:00.616091 22884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:05:00.616185 22884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:05:00.616312 22884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [term 1 FOLLOWER]: Advancing to term 2
I20260806 17:05:00.618400 22884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203b4541d648435b97f82aa04d720ecb" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43801 } }
I20260806 17:05:00.618736 22884 leader_election.cc:302] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 203b4541d648435b97f82aa04d720ecb; no voters:
I20260806 17:05:00.619091 22884 leader_election.cc:288] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 17:05:00.619228 22888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [term 2 FOLLOWER]: Leader election won for term 2
I20260806 17:05:00.619493 22888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [term 2 LEADER]: Becoming Leader. State: Replica: 203b4541d648435b97f82aa04d720ecb, State: Running, Role: LEADER
I20260806 17:05:00.619815 22888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203b4541d648435b97f82aa04d720ecb" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43801 } }
I20260806 17:05:00.620378 22884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:05:00.620847 22888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 203b4541d648435b97f82aa04d720ecb. Latest consensus state: current_term: 2 leader_uuid: "203b4541d648435b97f82aa04d720ecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203b4541d648435b97f82aa04d720ecb" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43801 } } }
I20260806 17:05:00.620960 22888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [sys.catalog]: This master's current role is: LEADER
I20260806 17:05:00.621245 22890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "203b4541d648435b97f82aa04d720ecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203b4541d648435b97f82aa04d720ecb" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 43801 } } }
I20260806 17:05:00.621382 22890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb [sys.catalog]: This master's current role is: LEADER
I20260806 17:05:00.621609 22892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:05:00.625067 22892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:05:00.625633 22884 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 17:05:00.625875 22892 catalog_manager.cc:1259] Loaded cluster ID: 3c0b22b40ede45889c99446dee19bf97
I20260806 17:05:00.625975 22892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:05:00.630955 22892 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:05:00.632579 22892 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb: Loaded TSK: 0
I20260806 17:05:00.634131 22892 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 17:05:00.739446 22886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:00.739828 22886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:00.739894 22886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:00.746340 22886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:00.746433 22886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:00.746529 22886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:05:00.754793 22886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.111.84:48429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:32795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=36693
--tserver_master_addrs=127.9.111.126:43801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:00.756291 22886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:00.757748 22886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:00.771163 22924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:00.771174 22921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:00.772552 22886 server_base.cc:1034] running on GCE node
W20260806 17:05:00.774695 22922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:00.775357 22886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:00.776276 22886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:00.777484 22886 hybrid_clock.cc:648] HybridClock initialized: now 1786035900777450 us; error 50 us; skew 500 ppm
Aug 06 17:05:00 dist-test-slave-69rg krb5kdc[21814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035900, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:00.781957 22886 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:05:00.783171 22886 webserver.cc:460] Webserver started at http://127.9.111.65:36693/ using document root <none> and password file <none>
I20260806 17:05:00.783936 22886 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:00.784065 22886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 17:05:00 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035900, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
I20260806 17:05:00.807171 22886 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 17:05:00.810130 22932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:00.811604 22886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 17:05:00.811754 22886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "1a907f91e7f74fa2b8542c44441c8fe7"
format_stamp: "Formatted at 2026-08-06 17:04:59 on dist-test-slave-69rg"
server_key: "20e1e672318ef581a3f347eacacdd744"
server_key_iv: "cd97598eec694006662287524182aa7a"
server_key_version: "kuduclusterkey@0"
I20260806 17:05:00.812474 22886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:00.825955 22886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:00.830046 22886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:00.830267 22886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:00.830996 22886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:05:00.832108 22886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:05:00.832178 22886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:00.832233 22886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:05:00.832262 22886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:00.844643 22886 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:32795
I20260806 17:05:00.847250 22886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:05:00.844662 23049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:32795 every 8 connection(s)
I20260806 17:05:00.856531 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 22886
I20260806 17:05:00.856959 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.111.84:48429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:35461
--local_ip_for_outbound_sockets=127.9.111.66
--tserver_master_addrs=127.9.111.126:43801
--webserver_port=33531
--webserver_interface=127.9.111.66
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.111.84:48429/kms with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 17:05:00 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035900, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:05:00.874516 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:00.850451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:60461 (local address 127.9.111.126:43801)
0806 17:05:00.851079 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:00.851104 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:00.854299 (+ 3195us) server_negotiation.cc:407] Connection header received
0806 17:05:00.856152 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:00.856173 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:00.856596 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:00.856890 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:00.858658 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:00.859890 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:00.861865 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:00.862788 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:00.862982 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:00.867054 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:00.867066 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:00.867092 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:00.867140 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:00.869687 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:00.870585 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:00.870596 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:00.870605 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:00.870817 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:00.871326 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:00.871333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:00.871337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:00.871908 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:00.872042 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:00.872845 (+ 803us) server_negotiation.cc:299] Negotiation successful
0806 17:05:00.873205 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":158,"threads_started":1}
I20260806 17:05:00.877029 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:00.851324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:43801 (local address 127.9.111.65:60461)
0806 17:05:00.854069 (+ 2745us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:00.854134 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:00.855514 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:00.857386 (+ 1872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:00.857393 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:00.857687 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:00.858417 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:00.858441 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:00.860104 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:00.860108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:00.861648 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:00.861665 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:00.863329 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:00.863336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:00.863588 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:00.864452 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:00.864484 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:00.866718 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:00.869932 (+ 3214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:00.869943 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:00.869964 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:00.870410 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:00.870961 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:00.870966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:00.870970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:00.871190 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:00.872114 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:00.872121 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:00.872447 (+ 326us) client_negotiation.cc:769] Sending connection context
0806 17:05:00.875563 (+ 3116us) client_negotiation.cc:240] Negotiation successful
0806 17:05:00.875965 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":156,"threads_started":1}
I20260806 17:05:00.880609 23050 heartbeater.cc:347] Connected to a master server at 127.9.111.126:43801
I20260806 17:05:00.880883 23050 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:00.881480 23050 heartbeater.cc:511] Master 127.9.111.126:43801 requested a full tablet report, sending...
W20260806 17:05:00.884289 23050 heartbeater.cc:644] Failed to heartbeat to 127.9.111.126:43801 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 17:05:00.884433 23050 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:00.884629 23050 heartbeater.cc:511] Master 127.9.111.126:43801 requested a full tablet report, sending...
I20260806 17:05:00.886015 23050 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:00.886214 23050 heartbeater.cc:511] Master 127.9.111.126:43801 requested a full tablet report, sending...
W20260806 17:05:00.887115 23050 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 17:05:01.018154 23054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:01.018508 23054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:01.018579 23054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:01.024129 23054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:01.024322 23054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:01.024495 23054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:05:01.034243 23054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.111.84:48429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:35461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=33531
--tserver_master_addrs=127.9.111.126:43801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:01.036203 23054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:01.037397 23054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:01.049610 23059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:01.049610 23060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:01.050561 23054 server_base.cc:1034] running on GCE node
W20260806 17:05:01.051383 23062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:01.051914 23054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:01.052678 23054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:01.060145 23054 hybrid_clock.cc:648] HybridClock initialized: now 1786035901060093 us; error 56 us; skew 500 ppm
Aug 06 17:05:01 dist-test-slave-69rg krb5kdc[21814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035901, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:01.063737 23054 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:05:01.064770 23054 webserver.cc:460] Webserver started at http://127.9.111.66:33531/ using document root <none> and password file <none>
I20260806 17:05:01.065405 23054 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:01.065534 23054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 17:05:01 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035901, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
I20260806 17:05:01.095822 23054 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 17:05:01.099306 23072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:01.100826 23054 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 17:05:01.100991 23054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "61742a45ecb0458aa19845e93cda241b"
format_stamp: "Formatted at 2026-08-06 17:04:59 on dist-test-slave-69rg"
server_key: "a32fc4f5b94ebdb73f9531cd8703bf9a"
server_key_iv: "232da25682469e5693fab80b8d869c0b"
server_key_version: "kuduclusterkey@0"
I20260806 17:05:01.101724 23054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:01.129334 23054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:01.133327 23054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:01.133562 23054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:01.134392 23054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:05:01.135619 23054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:05:01.135690 23054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:01.135761 23054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:05:01.135793 23054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:01.147436 23054 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:35461
I20260806 17:05:01.148825 23054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:05:01.152346 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 23054
I20260806 17:05:01.152886 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.111.84:48429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:41261
--local_ip_for_outbound_sockets=127.9.111.67
--tserver_master_addrs=127.9.111.126:43801
--webserver_port=35895
--webserver_interface=127.9.111.67
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.111.84:48429/kms with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:05:01.150357 23199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:35461 every 8 connection(s)
Aug 06 17:05:01 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035901, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:05:01.169986 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:01.151169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:43493 (local address 127.9.111.126:43801)
0806 17:05:01.151332 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:01.151338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:01.153970 (+ 2632us) server_negotiation.cc:407] Connection header received
0806 17:05:01.155557 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:01.155562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:01.155628 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:01.155731 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:01.157368 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:01.158297 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:01.160303 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:01.160765 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:01.160909 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:01.165063 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:01.165068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:01.165071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:01.165106 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:01.167119 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:01.167854 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:01.167858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:01.167860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:01.167953 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:01.168415 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:01.168420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:01.168422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:01.168696 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:01.168784 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:01.169605 (+ 821us) server_negotiation.cc:299] Negotiation successful
0806 17:05:01.169772 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 17:05:01.171794 23202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:01.151776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:43801 (local address 127.9.111.66:43493)
0806 17:05:01.153565 (+ 1789us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:01.153634 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:01.155246 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:01.156072 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:01.156083 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:01.156454 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:01.157168 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:01.157195 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:01.158471 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:01.158477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:01.160097 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:01.160114 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:01.160979 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:01.160984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:01.161218 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:01.163007 (+ 1789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:01.163030 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:01.164809 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:01.167296 (+ 2487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:01.167305 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:01.167327 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:01.167716 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:01.168068 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:01.168071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:01.168073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:01.168282 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:01.168879 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:01.168887 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:01.169283 (+ 396us) client_negotiation.cc:769] Sending connection context
0806 17:05:01.170282 (+ 999us) client_negotiation.cc:240] Negotiation successful
0806 17:05:01.170664 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1451,"thread_start_us":174,"threads_started":1}
I20260806 17:05:01.173252 23200 heartbeater.cc:347] Connected to a master server at 127.9.111.126:43801
I20260806 17:05:01.173516 23200 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:01.174168 23200 heartbeater.cc:511] Master 127.9.111.126:43801 requested a full tablet report, sending...
W20260806 17:05:01.176030 23200 heartbeater.cc:644] Failed to heartbeat to 127.9.111.126:43801 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 17:05:01.176177 23200 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:01.176391 23200 heartbeater.cc:511] Master 127.9.111.126:43801 requested a full tablet report, sending...
I20260806 17:05:01.177459 23200 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:01.177655 23200 heartbeater.cc:511] Master 127.9.111.126:43801 requested a full tablet report, sending...
W20260806 17:05:01.178516 23200 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 17:05:01.299449 23203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:01.299860 23203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:01.299948 23203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:01.305572 23203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:01.305672 23203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:01.305768 23203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:05:01.312716 23203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.111.84:48429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:41261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=35895
--tserver_master_addrs=127.9.111.126:43801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:01.313952 23203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:01.315023 23203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:01.327337 23213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:01.328423 23209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:01.329000 23210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:01.329615 23203 server_base.cc:1034] running on GCE node
I20260806 17:05:01.330154 23203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:01.330971 23203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:01.332181 23203 hybrid_clock.cc:648] HybridClock initialized: now 1786035901332147 us; error 44 us; skew 500 ppm
Aug 06 17:05:01 dist-test-slave-69rg krb5kdc[21814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035901, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:01.335455 23203 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:05:01.336566 23203 webserver.cc:460] Webserver started at http://127.9.111.67:35895/ using document root <none> and password file <none>
I20260806 17:05:01.337143 23203 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:01.337252 23203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 17:05:01 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035901, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
I20260806 17:05:01.363811 23203 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 17:05:01.371954 23222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:01.374138 23203 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260806 17:05:01.374305 23203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "11ce4c3e93df4518b231453bfe9f3577"
format_stamp: "Formatted at 2026-08-06 17:05:00 on dist-test-slave-69rg"
server_key: "aff7b7c4a8ed601b8a765f2f2700e1c8"
server_key_iv: "868e0ebdbb0f96d7fdc20224bb758f70"
server_key_version: "kuduclusterkey@0"
I20260806 17:05:01.375118 23203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:01.392688 23203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:01.396370 23203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:01.396600 23203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:01.397234 23203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:05:01.398254 23203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:05:01.398337 23203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:01.398408 23203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:05:01.398447 23203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:01.408603 23203 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:41261
I20260806 17:05:01.408890 23340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:41261 every 8 connection(s)
I20260806 17:05:01.409982 23203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:05:01.414278 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 23203
Aug 06 17:05:01 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035901, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:05:01.429237 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:01.414948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:34091 (local address 127.9.111.126:43801)
0806 17:05:01.415127 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:01.415132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:01.416061 (+ 929us) server_negotiation.cc:407] Connection header received
0806 17:05:01.417488 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:01.417492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:01.417551 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:01.417659 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:01.418879 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:01.419409 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:01.420600 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:01.420927 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:01.421041 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:01.424132 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:01.424136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:01.424139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:01.424166 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:01.425763 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:01.426822 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:01.426827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:01.426830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:01.426927 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:01.427400 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:01.427404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:01.427406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:01.427661 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:01.427738 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:01.428662 (+ 924us) server_negotiation.cc:299] Negotiation successful
0806 17:05:01.428824 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 17:05:01.430130 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:01.415404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:43801 (local address 127.9.111.67:34091)
0806 17:05:01.415901 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:01.415955 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:01.417213 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:01.417911 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:01.417923 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:01.418226 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:01.418716 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:01.418729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:01.419539 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:01.419542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:01.420427 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:01.420436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:01.421083 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:01.421087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:01.421262 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:01.422060 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:01.422081 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:01.423945 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:01.425949 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:01.425956 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:01.425971 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:01.426359 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:01.427060 (+ 701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:01.427063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:01.427066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:01.427269 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:01.427830 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:01.427836 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:01.428151 (+ 315us) client_negotiation.cc:769] Sending connection context
0806 17:05:01.428357 (+ 206us) client_negotiation.cc:240] Negotiation successful
0806 17:05:01.428712 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":144,"threads_started":1}
I20260806 17:05:01.431422 23341 heartbeater.cc:347] Connected to a master server at 127.9.111.126:43801
I20260806 17:05:01.431721 23341 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:01.432358 23341 heartbeater.cc:511] Master 127.9.111.126:43801 requested a full tablet report, sending...
I20260806 17:05:01.432528 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:01.416675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58414 (local address 127.9.111.126:43801)
0806 17:05:01.421705 (+ 5030us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:01.421710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:01.421728 (+ 18us) server_negotiation.cc:407] Connection header received
0806 17:05:01.421766 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:01.421769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:01.421855 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:01.422016 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:01.423050 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:01.423583 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:01.425089 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:01.426205 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:01.426314 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:01.428026 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:01.428031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:01.428033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:01.428061 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:01.429842 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:01.430501 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:01.430505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:01.430507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:01.430601 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:01.431479 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:01.431485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:01.431486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:01.431728 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:01.431949 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:01.432204 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 17:05:01.432352 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4946,"thread_start_us":93,"threads_started":1}
W20260806 17:05:01.434278 23341 heartbeater.cc:644] Failed to heartbeat to 127.9.111.126:43801 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 17:05:01.434401 23341 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:01.434563 23341 heartbeater.cc:511] Master 127.9.111.126:43801 requested a full tablet report, sending...
I20260806 17:05:01.435376 23341 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:01.435559 23341 heartbeater.cc:511] Master 127.9.111.126:43801 requested a full tablet report, sending...
W20260806 17:05:01.436277 23341 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 17:05:01 dist-test-slave-69rg krb5kdc[21814](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786035901, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:01.887403 23050 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:01.887753 23050 heartbeater.cc:511] Master 127.9.111.126:43801 requested a full tablet report, sending...
I20260806 17:05:01.889228 22846 ts_manager.cc:194] Registered new tserver with Master: 1a907f91e7f74fa2b8542c44441c8fe7 (127.9.111.65:32795)
I20260806 17:05:01.891006 22846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:60461
I20260806 17:05:02.178795 23200 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:02.179147 23200 heartbeater.cc:511] Master 127.9.111.126:43801 requested a full tablet report, sending...
I20260806 17:05:02.180392 22846 ts_manager.cc:194] Registered new tserver with Master: 61742a45ecb0458aa19845e93cda241b (127.9.111.66:35461)
I20260806 17:05:02.181253 22846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:43493
Aug 06 17:05:02 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035901, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
I20260806 17:05:02.436542 23341 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:02.436920 23341 heartbeater.cc:511] Master 127.9.111.126:43801 requested a full tablet report, sending...
I20260806 17:05:02.438262 22846 ts_manager.cc:194] Registered new tserver with Master: 11ce4c3e93df4518b231453bfe9f3577 (127.9.111.67:41261)
I20260806 17:05:02.439059 22846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:34091
I20260806 17:05:02.439440 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 17:05:02 dist-test-slave-69rg krb5kdc[21814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035901, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for HTTP/127.9.111.84@KRBTEST.COM
I20260806 17:05:02.461534 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:02.446595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58426 (local address 127.9.111.126:43801)
0806 17:05:02.447014 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:02.447019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:02.448005 (+ 986us) server_negotiation.cc:407] Connection header received
0806 17:05:02.448070 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:02.448074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:02.448131 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:02.448219 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:02.449362 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.453186 (+ 3824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.454450 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.454797 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.454918 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:02.456946 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:02.456951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:02.456954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:02.456992 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:02.459140 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:02.459783 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:02.459787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:02.459789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:02.459897 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:02.460224 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:02.460228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:02.460229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:02.460498 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:02.460643 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:02.460987 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 17:05:02.461144 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":111,"threads_started":1}
I20260806 17:05:02.471976 22846 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:05:02.478264 22846 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:05:02.503651 23371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:02.494933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:41261 (local address 127.0.0.1:48580)
0806 17:05:02.497177 (+ 2244us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:02.497250 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:02.497405 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:02.498324 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:02.498331 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:02.498358 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:02.498524 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:02.498530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.500105 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.500110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:02.501540 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:02.501555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.502951 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.502955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:02.503147 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:02.503210 (+ 63us) client_negotiation.cc:769] Sending connection context
0806 17:05:02.503322 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 17:05:02.503461 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2052,"mutex_wait_us":41,"thread_start_us":141,"threads_started":1}
I20260806 17:05:02.504323 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:02.495915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.9.111.67:41261)
0806 17:05:02.497707 (+ 1792us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:02.497714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:02.497748 (+ 34us) server_negotiation.cc:407] Connection header received
0806 17:05:02.497865 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:02.497871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:02.498134 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:02.498681 (+ 547us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:02.498829 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.499976 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.501700 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.502587 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.502722 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:02.502889 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:02.504055 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0806 17:05:02.504175 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1681,"thread_start_us":105,"threads_started":1}
I20260806 17:05:02.507809 23270 tablet_service.cc:1467] Processing CreateTablet for tablet 2a05d096631940eb9373401a275b8d53 (DEFAULT_TABLE table=test-table [id=d1df5f27ca4f4eaaad155f2490479cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:05:02.509120 23270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a05d096631940eb9373401a275b8d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:02.510578 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:02.496030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60808 (local address 127.9.111.66:35461)
0806 17:05:02.496901 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:02.496910 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:02.502165 (+ 5255us) server_negotiation.cc:407] Connection header received
0806 17:05:02.502316 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:02.502323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:02.502594 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:02.502755 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:02.503238 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.504330 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.507311 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.508176 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.508274 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:02.508427 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:02.510284 (+ 1857us) server_negotiation.cc:299] Negotiation successful
0806 17:05:02.510419 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":86,"threads_started":1}
I20260806 17:05:02.511284 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:02.495359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:35461 (local address 127.0.0.1:60808)
0806 17:05:02.502010 (+ 6651us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:02.502028 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:02.502168 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:02.502922 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:02.502926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:02.502947 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:02.503078 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:02.503084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.505936 (+ 2852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.505940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:02.507107 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:02.507120 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.508688 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.508692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:02.508809 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:02.508834 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 17:05:02.511017 (+ 2183us) client_negotiation.cc:240] Negotiation successful
0806 17:05:02.511119 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6564,"mutex_wait_us":6,"thread_start_us":107,"threads_started":1}
I20260806 17:05:02.512027 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:02.496893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:32795 (local address 127.0.0.1:46092)
0806 17:05:02.506351 (+ 9458us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:02.506383 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:02.506698 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:02.507555 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:02.507558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:02.507578 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:02.507677 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:02.507683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.509065 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.509068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:02.510224 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:02.510248 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.511599 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.511603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:02.511701 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:02.511728 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 17:05:02.511810 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 17:05:02.511887 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9296,"thread_start_us":115,"threads_started":1}
I20260806 17:05:02.514917 23104 tablet_service.cc:1467] Processing CreateTablet for tablet 2a05d096631940eb9373401a275b8d53 (DEFAULT_TABLE table=test-table [id=d1df5f27ca4f4eaaad155f2490479cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:05:02.516330 23104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a05d096631940eb9373401a275b8d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:02.522410 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:02.497341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.9.111.65:32795)
0806 17:05:02.500300 (+ 2959us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:02.500307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:02.506539 (+ 6232us) server_negotiation.cc:407] Connection header received
0806 17:05:02.506918 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:02.506927 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:02.507205 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:02.507355 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:02.507822 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.508925 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.510395 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.511229 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.511359 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:02.511520 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:02.522063 (+ 10543us) server_negotiation.cc:299] Negotiation successful
0806 17:05:02.522227 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2829,"thread_start_us":80,"threads_started":1}
I20260806 17:05:02.523311 23376 tablet_bootstrap.cc:492] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577: Bootstrap starting.
I20260806 17:05:02.526707 23376 tablet_bootstrap.cc:654] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:02.526758 22962 tablet_service.cc:1467] Processing CreateTablet for tablet 2a05d096631940eb9373401a275b8d53 (DEFAULT_TABLE table=test-table [id=d1df5f27ca4f4eaaad155f2490479cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:05:02.528524 23376 log.cc:824] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:02.529762 22962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a05d096631940eb9373401a275b8d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:02.544348 23376 tablet_bootstrap.cc:492] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577: No bootstrap required, opened a new log
I20260806 17:05:02.544648 23376 ts_tablet_manager.cc:1397] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20260806 17:05:02.548295 23376 raft_consensus.cc:348] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ce4c3e93df4518b231453bfe9f3577" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41261 } } peers { permanent_uuid: "61742a45ecb0458aa19845e93cda241b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 35461 } } peers { permanent_uuid: "1a907f91e7f74fa2b8542c44441c8fe7" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 32795 } }
I20260806 17:05:02.548687 23376 raft_consensus.cc:374] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:02.548791 23376 raft_consensus.cc:729] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11ce4c3e93df4518b231453bfe9f3577, State: Initialized, Role: FOLLOWER
I20260806 17:05:02.549378 23376 consensus_queue.cc:260] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ce4c3e93df4518b231453bfe9f3577" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41261 } } peers { permanent_uuid: "61742a45ecb0458aa19845e93cda241b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 35461 } } peers { permanent_uuid: "1a907f91e7f74fa2b8542c44441c8fe7" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 32795 } }
I20260806 17:05:02.550928 23376 ts_tablet_manager.cc:1428] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 17:05:02.551414 23341 heartbeater.cc:503] Master 127.9.111.126:43801 was elected leader, sending a full tablet report...
I20260806 17:05:02.554116 23381 tablet_bootstrap.cc:492] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b: Bootstrap starting.
I20260806 17:05:02.557291 23381 tablet_bootstrap.cc:654] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:02.559412 23379 tablet_bootstrap.cc:492] T 2a05d096631940eb9373401a275b8d53 P 1a907f91e7f74fa2b8542c44441c8fe7: Bootstrap starting.
I20260806 17:05:02.560662 23381 log.cc:824] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:02.562095 23379 tablet_bootstrap.cc:654] T 2a05d096631940eb9373401a275b8d53 P 1a907f91e7f74fa2b8542c44441c8fe7: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:02.563192 23379 log.cc:824] T 2a05d096631940eb9373401a275b8d53 P 1a907f91e7f74fa2b8542c44441c8fe7: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:02.566579 23379 tablet_bootstrap.cc:492] T 2a05d096631940eb9373401a275b8d53 P 1a907f91e7f74fa2b8542c44441c8fe7: No bootstrap required, opened a new log
I20260806 17:05:02.566838 23379 ts_tablet_manager.cc:1397] T 2a05d096631940eb9373401a275b8d53 P 1a907f91e7f74fa2b8542c44441c8fe7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 17:05:02.567224 23381 tablet_bootstrap.cc:492] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b: No bootstrap required, opened a new log
I20260806 17:05:02.567477 23381 ts_tablet_manager.cc:1397] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.002s
I20260806 17:05:02.572697 23379 raft_consensus.cc:348] T 2a05d096631940eb9373401a275b8d53 P 1a907f91e7f74fa2b8542c44441c8fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ce4c3e93df4518b231453bfe9f3577" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41261 } } peers { permanent_uuid: "61742a45ecb0458aa19845e93cda241b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 35461 } } peers { permanent_uuid: "1a907f91e7f74fa2b8542c44441c8fe7" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 32795 } }
I20260806 17:05:02.572685 23381 raft_consensus.cc:348] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ce4c3e93df4518b231453bfe9f3577" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41261 } } peers { permanent_uuid: "61742a45ecb0458aa19845e93cda241b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 35461 } } peers { permanent_uuid: "1a907f91e7f74fa2b8542c44441c8fe7" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 32795 } }
I20260806 17:05:02.573211 23379 raft_consensus.cc:374] T 2a05d096631940eb9373401a275b8d53 P 1a907f91e7f74fa2b8542c44441c8fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:02.573228 23381 raft_consensus.cc:374] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:02.573297 23381 raft_consensus.cc:729] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61742a45ecb0458aa19845e93cda241b, State: Initialized, Role: FOLLOWER
I20260806 17:05:02.573341 23379 raft_consensus.cc:729] T 2a05d096631940eb9373401a275b8d53 P 1a907f91e7f74fa2b8542c44441c8fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a907f91e7f74fa2b8542c44441c8fe7, State: Initialized, Role: FOLLOWER
I20260806 17:05:02.573972 23379 consensus_queue.cc:260] T 2a05d096631940eb9373401a275b8d53 P 1a907f91e7f74fa2b8542c44441c8fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ce4c3e93df4518b231453bfe9f3577" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41261 } } peers { permanent_uuid: "61742a45ecb0458aa19845e93cda241b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 35461 } } peers { permanent_uuid: "1a907f91e7f74fa2b8542c44441c8fe7" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 32795 } }
I20260806 17:05:02.573973 23381 consensus_queue.cc:260] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ce4c3e93df4518b231453bfe9f3577" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41261 } } peers { permanent_uuid: "61742a45ecb0458aa19845e93cda241b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 35461 } } peers { permanent_uuid: "1a907f91e7f74fa2b8542c44441c8fe7" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 32795 } }
I20260806 17:05:02.575296 23379 ts_tablet_manager.cc:1428] T 2a05d096631940eb9373401a275b8d53 P 1a907f91e7f74fa2b8542c44441c8fe7: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 17:05:02.575299 23381 ts_tablet_manager.cc:1428] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260806 17:05:02.575698 23200 heartbeater.cc:503] Master 127.9.111.126:43801 was elected leader, sending a full tablet report...
I20260806 17:05:02.575737 23050 heartbeater.cc:503] Master 127.9.111.126:43801 was elected leader, sending a full tablet report...
W20260806 17:05:02.600025 23051 tablet.cc:2369] T 2a05d096631940eb9373401a275b8d53 P 1a907f91e7f74fa2b8542c44441c8fe7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:05:02.614552 23384 raft_consensus.cc:482] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:05:02.614830 23384 raft_consensus.cc:504] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ce4c3e93df4518b231453bfe9f3577" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41261 } } peers { permanent_uuid: "61742a45ecb0458aa19845e93cda241b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 35461 } } peers { permanent_uuid: "1a907f91e7f74fa2b8542c44441c8fe7" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 32795 } }
I20260806 17:05:02.617856 23384 leader_election.cc:288] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11ce4c3e93df4518b231453bfe9f3577 (127.9.111.67:41261), 1a907f91e7f74fa2b8542c44441c8fe7 (127.9.111.65:32795)
I20260806 17:05:02.622947 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:02.617216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:38243 (local address 127.9.111.67:41261)
0806 17:05:02.617342 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:02.617347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:02.617363 (+ 16us) server_negotiation.cc:407] Connection header received
0806 17:05:02.617420 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:02.617423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:02.617469 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:02.617553 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:02.618874 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.619668 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.621424 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.622107 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.622217 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:02.622273 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:02.622692 (+ 419us) server_negotiation.cc:299] Negotiation successful
0806 17:05:02.622799 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 17:05:02.627166 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:02.616488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:41261 (local address 127.9.111.66:38243)
0806 17:05:02.616939 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:02.616958 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:02.617180 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:02.617623 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:02.617626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:02.617649 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:02.618696 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:02.618718 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.619843 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.619846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:02.621260 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:02.621274 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.622421 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.622425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:02.622541 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:02.622565 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 17:05:02.626836 (+ 4271us) client_negotiation.cc:240] Negotiation successful
0806 17:05:02.626979 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":125,"threads_started":1}
I20260806 17:05:02.628064 23292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a05d096631940eb9373401a275b8d53" candidate_uuid: "61742a45ecb0458aa19845e93cda241b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11ce4c3e93df4518b231453bfe9f3577" is_pre_election: true
I20260806 17:05:02.628430 23292 raft_consensus.cc:2454] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61742a45ecb0458aa19845e93cda241b in term 0.
I20260806 17:05:02.629163 23074 leader_election.cc:302] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11ce4c3e93df4518b231453bfe9f3577, 61742a45ecb0458aa19845e93cda241b; no voters:
I20260806 17:05:02.629655 23384 raft_consensus.cc:2781] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:05:02.629761 23384 raft_consensus.cc:482] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:05:02.629804 23384 raft_consensus.cc:3037] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:02.631222 23384 raft_consensus.cc:504] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ce4c3e93df4518b231453bfe9f3577" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41261 } } peers { permanent_uuid: "61742a45ecb0458aa19845e93cda241b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 35461 } } peers { permanent_uuid: "1a907f91e7f74fa2b8542c44441c8fe7" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 32795 } }
I20260806 17:05:02.632253 23292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a05d096631940eb9373401a275b8d53" candidate_uuid: "61742a45ecb0458aa19845e93cda241b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11ce4c3e93df4518b231453bfe9f3577"
I20260806 17:05:02.632310 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:02.618557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:52043 (local address 127.9.111.65:32795)
0806 17:05:02.618703 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:02.618708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:02.627663 (+ 8955us) server_negotiation.cc:407] Connection header received
0806 17:05:02.627931 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:02.627935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:02.627991 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:02.628045 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:02.628439 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.629209 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.630787 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.631377 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.631486 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:02.631542 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:02.632052 (+ 510us) server_negotiation.cc:299] Negotiation successful
0806 17:05:02.632147 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 17:05:02.632404 23292 raft_consensus.cc:3037] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:02.632673 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:02.618062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:32795 (local address 127.9.111.66:52043)
0806 17:05:02.627573 (+ 9511us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:02.627589 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:02.627776 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:02.628197 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:02.628200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:02.628219 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:02.628312 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:02.628317 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.629362 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.629365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:02.630620 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:02.630631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.631802 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.631806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:02.631904 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:02.631928 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 17:05:02.632463 (+ 535us) client_negotiation.cc:240] Negotiation successful
0806 17:05:02.632548 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9423,"thread_start_us":134,"threads_started":1}
I20260806 17:05:02.633792 23292 raft_consensus.cc:2454] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61742a45ecb0458aa19845e93cda241b in term 1.
I20260806 17:05:02.634202 23074 leader_election.cc:302] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11ce4c3e93df4518b231453bfe9f3577, 61742a45ecb0458aa19845e93cda241b; no voters:
I20260806 17:05:02.634572 23388 raft_consensus.cc:2781] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:05:02.634627 23002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a05d096631940eb9373401a275b8d53" candidate_uuid: "61742a45ecb0458aa19845e93cda241b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a907f91e7f74fa2b8542c44441c8fe7" is_pre_election: true
I20260806 17:05:02.634769 23388 raft_consensus.cc:686] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [term 1 LEADER]: Becoming Leader. State: Replica: 61742a45ecb0458aa19845e93cda241b, State: Running, Role: LEADER
I20260806 17:05:02.634976 23002 raft_consensus.cc:2454] T 2a05d096631940eb9373401a275b8d53 P 1a907f91e7f74fa2b8542c44441c8fe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61742a45ecb0458aa19845e93cda241b in term 0.
I20260806 17:05:02.635143 23388 consensus_queue.cc:237] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ce4c3e93df4518b231453bfe9f3577" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41261 } } peers { permanent_uuid: "61742a45ecb0458aa19845e93cda241b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 35461 } } peers { permanent_uuid: "1a907f91e7f74fa2b8542c44441c8fe7" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 32795 } }
I20260806 17:05:02.638036 23002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a05d096631940eb9373401a275b8d53" candidate_uuid: "61742a45ecb0458aa19845e93cda241b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a907f91e7f74fa2b8542c44441c8fe7"
I20260806 17:05:02.638219 23002 raft_consensus.cc:3037] T 2a05d096631940eb9373401a275b8d53 P 1a907f91e7f74fa2b8542c44441c8fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:02.639077 22846 catalog_manager.cc:5495] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b reported cstate change: term changed from 0 to 1, leader changed from <none> to 61742a45ecb0458aa19845e93cda241b (127.9.111.66). New cstate: current_term: 1 leader_uuid: "61742a45ecb0458aa19845e93cda241b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ce4c3e93df4518b231453bfe9f3577" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61742a45ecb0458aa19845e93cda241b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 35461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a907f91e7f74fa2b8542c44441c8fe7" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 32795 } health_report { overall_health: UNKNOWN } } }
I20260806 17:05:02.639798 23002 raft_consensus.cc:2454] T 2a05d096631940eb9373401a275b8d53 P 1a907f91e7f74fa2b8542c44441c8fe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61742a45ecb0458aa19845e93cda241b in term 1.
I20260806 17:05:02.640471 23384 leader_election.cc:288] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [CANDIDATE]: Term 1 election: Requested vote from peers 11ce4c3e93df4518b231453bfe9f3577 (127.9.111.67:41261), 1a907f91e7f74fa2b8542c44441c8fe7 (127.9.111.65:32795)
W20260806 17:05:02.652175 23201 tablet.cc:2369] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:05:02.665108 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:02.659478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60820 (local address 127.9.111.66:35461)
0806 17:05:02.659628 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:02.659633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:02.659649 (+ 16us) server_negotiation.cc:407] Connection header received
0806 17:05:02.659751 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:02.659754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:02.659830 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:02.659960 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:05:02.660420 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.661201 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.662723 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:02.663072 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:02.663173 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:02.663639 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:05:02.664241 (+ 602us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:05:02.664385 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:02.664872 (+ 487us) server_negotiation.cc:299] Negotiation successful
0806 17:05:02.664937 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 17:05:02.668401 23342 tablet.cc:2369] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:05:02.673040 23002 raft_consensus.cc:1261] T 2a05d096631940eb9373401a275b8d53 P 1a907f91e7f74fa2b8542c44441c8fe7 [term 1 FOLLOWER]: Refusing update from remote peer 61742a45ecb0458aa19845e93cda241b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:05:02.674145 23292 raft_consensus.cc:1261] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577 [term 1 FOLLOWER]: Refusing update from remote peer 61742a45ecb0458aa19845e93cda241b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:05:02.674664 23389 consensus_queue.cc:1035] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a907f91e7f74fa2b8542c44441c8fe7" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:05:02.674880 23388 consensus_queue.cc:1035] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [LEADER]: Connected to new peer: Peer: permanent_uuid: "11ce4c3e93df4518b231453bfe9f3577" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:05:02.692873 23392 mvcc.cc:205] Tried to move back new op lower bound from 7315603057340297216 to 7315603057198264320. Current Snapshot: MvccSnapshot[applied={T|T < 7315603057340297216 or (T in {7315603057340297216})}]
I20260806 17:05:02.693333 23393 mvcc.cc:205] Tried to move back new op lower bound from 7315603057340297216 to 7315603057198264320. Current Snapshot: MvccSnapshot[applied={T|T < 7315603057340297216}]
I20260806 17:05:02.693621 23394 mvcc.cc:205] Tried to move back new op lower bound from 7315603057340297216 to 7315603057198264320. Current Snapshot: MvccSnapshot[applied={T|T < 7315603057340297216}]
I20260806 17:05:02.700778 22845 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:05:02.700973 22845 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:05:02.704130 22845 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 203b4541d648435b97f82aa04d720ecb: Sending DeleteTablet for 3 replicas of tablet 2a05d096631940eb9373401a275b8d53
I20260806 17:05:02.705096 23270 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a05d096631940eb9373401a275b8d53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:05:02 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:48580
I20260806 17:05:02.706279 23104 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a05d096631940eb9373401a275b8d53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:05:02 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:60808
I20260806 17:05:02.707340 22962 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a05d096631940eb9373401a275b8d53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:05:02 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:46092
I20260806 17:05:02.708488 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 22886
I20260806 17:05:02.712745 23402 tablet_replica.cc:330] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b: stopping tablet replica
I20260806 17:05:02.713218 23402 raft_consensus.cc:2229] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:05:02.713800 23402 raft_consensus.cc:2258] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:05:02.714550 23401 tablet_replica.cc:330] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577: stopping tablet replica
I20260806 17:05:02.714956 23401 raft_consensus.cc:2229] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:05:02.715777 23402 ts_tablet_manager.cc:1905] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:05:02.717389 23401 raft_consensus.cc:2258] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:05:02.719666 23402 ts_tablet_manager.cc:1918] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:05:02.719909 23402 log.cc:1196] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/2a05d096631940eb9373401a275b8d53
I20260806 17:05:02.720376 23402 ts_tablet_manager.cc:1939] T 2a05d096631940eb9373401a275b8d53 P 61742a45ecb0458aa19845e93cda241b: Deleting consensus metadata
I20260806 17:05:02.722051 22833 catalog_manager.cc:4841] TS 61742a45ecb0458aa19845e93cda241b (127.9.111.66:35461): tablet 2a05d096631940eb9373401a275b8d53 (table test-table [id=d1df5f27ca4f4eaaad155f2490479cc9]) successfully deleted
I20260806 17:05:02.730567 23401 ts_tablet_manager.cc:1905] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:05:02.733489 23401 ts_tablet_manager.cc:1918] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:05:02.733765 23401 log.cc:1196] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/2a05d096631940eb9373401a275b8d53
I20260806 17:05:02.734359 23401 ts_tablet_manager.cc:1939] T 2a05d096631940eb9373401a275b8d53 P 11ce4c3e93df4518b231453bfe9f3577: Deleting consensus metadata
W20260806 17:05:02.735930 22829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.111.65:32795 (error 108)
I20260806 17:05:02.735930 22830 catalog_manager.cc:4841] TS 11ce4c3e93df4518b231453bfe9f3577 (127.9.111.67:41261): tablet 2a05d096631940eb9373401a275b8d53 (table test-table [id=d1df5f27ca4f4eaaad155f2490479cc9]) successfully deleted
I20260806 17:05:02.736982 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 23054
I20260806 17:05:02.741997 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:02.741567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:32795 (local address 127.0.0.1:46094)
0806 17:05:02.741772 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:02.741893 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.65:32795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260806 17:05:02.742317 22829 catalog_manager.cc:4584] TS 1a907f91e7f74fa2b8542c44441c8fe7 (127.9.111.65:32795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a05d096631940eb9373401a275b8d53: Network error: Client connection negotiation failed: client connection to 127.9.111.65:32795: connect: Connection refused (error 111)
I20260806 17:05:02.747906 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 23203
I20260806 17:05:02.757004 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 22812
I20260806 17:05:02.765466 9661 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 17:05:03.111542 9661 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 17:05:03.111675 9661 mini_ranger.cc:66] Stopping Ranger...
I20260806 17:05:03.458009 9661 mini_ranger.cc:68] Stopped Ranger
2026-08-06 17:05:03.458 UTC [21845] LOG: received smart shutdown request
2026-08-06 17:05:03.463 UTC [21845] LOG: background worker "logical replication launcher" (PID 21855) exited with exit code 1
2026-08-06 17:05:03.463 UTC [21850] LOG: shutting down
2026-08-06 17:05:03.679 UTC [21845] LOG: database system is shut down
2026-08-06T17:05:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53261 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:05:03 dist-test-slave-69rg krb5kdc[23415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:05:03 dist-test-slave-69rg krb5kdc[23415](info): set up 2 sockets
Aug 06 17:05:03 dist-test-slave-69rg krb5kdc[23415](info): commencing operation
krb5kdc: starting...
W20260806 17:05:04.886194 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:05:04 dist-test-slave-69rg krb5kdc[23415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:05:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:05:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:05:05.026885 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:42607
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:42607,127.9.111.125:40811,127.9.111.124:42953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:05:05.146554 23429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:05.146853 23429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:05.146896 23429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:05.151274 23429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:05:05.151360 23429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:05.151381 23429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:05.151398 23429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:05:05.151414 23429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:05:05.157233 23429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:42607,127.9.111.125:40811,127.9.111.124:42953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:42607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:05.158488 23429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:05.159519 23429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:05.169098 23434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:05.169100 23437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:05.169209 23435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:05.169317 23429 server_base.cc:1034] running on GCE node
I20260806 17:05:05.170032 23429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:05.171293 23429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:05.172569 23429 hybrid_clock.cc:648] HybridClock initialized: now 1786035905172553 us; error 38 us; skew 500 ppm
Aug 06 17:05:05 dist-test-slave-69rg krb5kdc[23415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035905, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:05.175894 23429 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:05:05.176918 23429 webserver.cc:460] Webserver started at http://127.9.111.126:35375/ using document root <none> and password file <none>
I20260806 17:05:05.177505 23429 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:05.177603 23429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:05.177903 23429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:05:05.180042 23429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "284269db8738422b81edcce2505564c6"
format_stamp: "Formatted at 2026-08-06 17:05:05 on dist-test-slave-69rg"
server_key: "7f64bbb7f80b779a9802f1bbef66e1f0"
server_key_iv: "c50ac2c4f7d46ac464b44d9556cfb160"
server_key_version: "encryptionkey@0"
I20260806 17:05:05.180658 23429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "284269db8738422b81edcce2505564c6"
format_stamp: "Formatted at 2026-08-06 17:05:05 on dist-test-slave-69rg"
server_key: "7f64bbb7f80b779a9802f1bbef66e1f0"
server_key_iv: "c50ac2c4f7d46ac464b44d9556cfb160"
server_key_version: "encryptionkey@0"
I20260806 17:05:05.184573 23429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260806 17:05:05.187212 23445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:05.188511 23429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 17:05:05.188678 23429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "284269db8738422b81edcce2505564c6"
format_stamp: "Formatted at 2026-08-06 17:05:05 on dist-test-slave-69rg"
server_key: "7f64bbb7f80b779a9802f1bbef66e1f0"
server_key_iv: "c50ac2c4f7d46ac464b44d9556cfb160"
server_key_version: "encryptionkey@0"
I20260806 17:05:05.188800 23429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:05.214468 23429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:05.218123 23429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:05.218412 23429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:05.226608 23429 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:42607
I20260806 17:05:05.226625 23501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:42607 every 8 connection(s)
I20260806 17:05:05.227648 23429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:05:05.230782 23503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:05.235551 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 23429
I20260806 17:05:05.235314 23503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } has no permanent_uuid. Determining permanent_uuid...
I20260806 17:05:05.235733 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:05:05.236042 9661 external_mini_cluster.cc:1428] Setting key 554e919dd2215db0b228db91c54ccbda
WARNING: no policy specified for kudu/127.9.111.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.125@KRBTEST.COM" created.
Aug 06 17:05:05 dist-test-slave-69rg krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035905, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:05:05.256045 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.238323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:42607 (local address 127.0.0.1:46692)
0806 17:05:05.238866 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.238921 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:05.239804 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:05.240577 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:05.240582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:05.241056 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:05.241323 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:05.241346 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.242639 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.242643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:05.243417 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:05.243426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.244110 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.244114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:05.244228 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.244940 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:05.244958 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:05.250158 (+ 5200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:05.252646 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:05.252657 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:05.252667 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:05.252962 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:05.253340 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:05.253343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:05.253345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:05.253487 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:05.254225 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:05.254232 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:05.254375 (+ 143us) client_negotiation.cc:769] Sending connection context
0806 17:05:05.254560 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 17:05:05.254985 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":107,"thread_start_us":155,"threads_started":1}
I20260806 17:05:05.256057 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.238324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46692 (local address 127.9.111.126:42607)
0806 17:05:05.238945 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:05.238958 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:05.239041 (+ 83us) server_negotiation.cc:407] Connection header received
0806 17:05:05.240162 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:05.240173 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:05.240370 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:05.240580 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:05.241502 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.242502 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.243552 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.243941 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.244122 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.250396 (+ 6274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:05.250407 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:05.250427 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:05.250471 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:05.252505 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.253092 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.253099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.253101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.253232 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.253592 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.253596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.253598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.254101 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:05.254247 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:05.254622 (+ 375us) server_negotiation.cc:299] Negotiation successful
0806 17:05:05.254898 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":188,"threads_started":1}
I20260806 17:05:05.258836 23503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } has no permanent_uuid. Determining permanent_uuid...
I20260806 17:05:05.259709 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.259424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:40811 (local address 127.0.0.1:58424)
0806 17:05:05.259556 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.259635 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:40811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 17:05:05.259950 23446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.111.125:40811: connect: Connection refused (error 111)
I20260806 17:05:05.260991 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.260739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:40811 (local address 127.0.0.1:58430)
0806 17:05:05.260867 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.260918 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:40811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 17:05:05.261189 23503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.125:40811: Network error: Client connection negotiation failed: client connection to 127.9.111.125:40811: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.125.keytab.
Entry for principal kudu/127.9.111.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.125.keytab.
I20260806 17:05:05.270256 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.125:40811
--webserver_interface=127.9.111.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.125.keytab
--principal=kudu/127.9.111.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:42607,127.9.111.125:40811,127.9.111.124:42953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:05:05.310510 23503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } attempt: 1
I20260806 17:05:05.311553 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.311226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:40811 (local address 127.0.0.1:58432)
0806 17:05:05.311395 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.311470 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:40811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 17:05:05.312359 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.312150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:40811 (local address 127.0.0.1:58440)
0806 17:05:05.312255 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.312303 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:40811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 17:05:05.312664 23503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.125:40811: Network error: Client connection negotiation failed: client connection to 127.9.111.125:40811: connect: Connection refused (error 111)
I20260806 17:05:05.380856 23503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } attempt: 2
I20260806 17:05:05.381748 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.381447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:40811 (local address 127.0.0.1:58456)
0806 17:05:05.381598 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.381668 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:40811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 17:05:05.382556 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.382350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:40811 (local address 127.0.0.1:58468)
0806 17:05:05.382448 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.382495 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:40811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 17:05:05.382704 23503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.125:40811: Network error: Client connection negotiation failed: client connection to 127.9.111.125:40811: connect: Connection refused (error 111)
W20260806 17:05:05.388806 23510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:05.389123 23510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:05.389168 23510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:05.393553 23510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:05:05.393630 23510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:05.393651 23510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:05.393669 23510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:05:05.393685 23510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:05:05.399437 23510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:42607,127.9.111.125:40811,127.9.111.124:42953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.125.keytab
--principal=kudu/127.9.111.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.125:40811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.111.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:05.400671 23510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:05.401722 23510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:05.410575 23516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:05.410564 23515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:05.410637 23518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:05.410862 23510 server_base.cc:1034] running on GCE node
I20260806 17:05:05.411696 23510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:05.412899 23510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:05.414090 23510 hybrid_clock.cc:648] HybridClock initialized: now 1786035905414057 us; error 46 us; skew 500 ppm
Aug 06 17:05:05 dist-test-slave-69rg krb5kdc[23415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035905, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:05.417174 23510 init.cc:377] Logged in from keytab as kudu/127.9.111.125@KRBTEST.COM (short username kudu)
I20260806 17:05:05.418212 23510 webserver.cc:460] Webserver started at http://127.9.111.125:45135/ using document root <none> and password file <none>
I20260806 17:05:05.418790 23510 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:05.418891 23510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:05.419144 23510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:05:05.421270 23510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-1/data/instance:
uuid: "cf0ee50c23da4c13a30e08b4d437df1a"
format_stamp: "Formatted at 2026-08-06 17:05:05 on dist-test-slave-69rg"
server_key: "e18d91789a392025b035cbcf42a2dfcb"
server_key_iv: "ae2f072cd31bd53c987e6298f27514ee"
server_key_version: "encryptionkey@0"
I20260806 17:05:05.421934 23510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-1/wal/instance:
uuid: "cf0ee50c23da4c13a30e08b4d437df1a"
format_stamp: "Formatted at 2026-08-06 17:05:05 on dist-test-slave-69rg"
server_key: "e18d91789a392025b035cbcf42a2dfcb"
server_key_iv: "ae2f072cd31bd53c987e6298f27514ee"
server_key_version: "encryptionkey@0"
I20260806 17:05:05.425905 23510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 17:05:05.428457 23525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:05.429945 23510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 17:05:05.430083 23510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-1/wal
uuid: "cf0ee50c23da4c13a30e08b4d437df1a"
format_stamp: "Formatted at 2026-08-06 17:05:05 on dist-test-slave-69rg"
server_key: "e18d91789a392025b035cbcf42a2dfcb"
server_key_iv: "ae2f072cd31bd53c987e6298f27514ee"
server_key_version: "encryptionkey@0"
I20260806 17:05:05.430210 23510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:05.445452 23510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:05.449594 23510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:05.449920 23510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:05.457911 23510 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.125:40811
I20260806 17:05:05.458001 23578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.125:40811 every 8 connection(s)
I20260806 17:05:05.459034 23510 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-1/data/info.pb
I20260806 17:05:05.462134 23579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:05.466123 23579 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } has no permanent_uuid. Determining permanent_uuid...
I20260806 17:05:05.468531 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 23510
I20260806 17:05:05.468681 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-1/wal/instance
I20260806 17:05:05.469046 9661 external_mini_cluster.cc:1428] Setting key cba7bb52b0130a0f9a1fe1e56888f5e1
I20260806 17:05:05.473878 23503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } attempt: 3
Aug 06 17:05:05 dist-test-slave-69rg krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035905, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.125@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.111.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.124@KRBTEST.COM" created.
Aug 06 17:05:05 dist-test-slave-69rg krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035905, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for kudu/127.9.111.125@KRBTEST.COM
I20260806 17:05:05.484498 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.468124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46706 (local address 127.9.111.126:42607)
0806 17:05:05.468313 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:05.468318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:05.469480 (+ 1162us) server_negotiation.cc:407] Connection header received
0806 17:05:05.470542 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:05.470546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:05.470614 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:05.470749 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:05.472585 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.473500 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.475321 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.475593 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.475702 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.479134 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:05.479137 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:05.479139 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:05.479188 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:05.481510 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.482237 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.482241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.482242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.482326 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.482875 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.482882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.482887 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.483249 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:05.483336 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:05.484046 (+ 710us) server_negotiation.cc:299] Negotiation successful
0806 17:05:05.484218 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 17:05:05.485846 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.468678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:42607 (local address 127.0.0.1:46706)
0806 17:05:05.469315 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.469363 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:05.470314 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:05.470939 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:05.470952 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:05.471511 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:05.472321 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:05.472348 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.473710 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.473713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:05.475177 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:05.475189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.475830 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.475837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:05.476108 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.476877 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:05.476904 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:05.478935 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:05.481654 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:05.481660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:05.481674 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:05.482124 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:05.482465 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:05.482469 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:05.482471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:05.482734 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:05.483401 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:05.483406 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:05.483721 (+ 315us) client_negotiation.cc:769] Sending connection context
0806 17:05:05.483948 (+ 227us) client_negotiation.cc:240] Negotiation successful
0806 17:05:05.484640 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":184,"threads_started":1}
I20260806 17:05:05.487367 23579 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } has no permanent_uuid. Determining permanent_uuid...
I20260806 17:05:05.488054 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.474398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:40811 (local address 127.0.0.1:58472)
0806 17:05:05.474600 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.474716 (+ 116us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:05.474822 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:05.475595 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:05.475599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:05.475831 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:05.475921 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:05.475927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.477091 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.477094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:05.477632 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:05.477641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.478558 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.478562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:05.478674 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.479224 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:05.479242 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:05.483575 (+ 4333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:05.486071 (+ 2496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:05.486075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:05.486076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:05.486351 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:05.486708 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:05.486712 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:05.486714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:05.486810 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:05.487462 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:05.487469 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:05.487583 (+ 114us) client_negotiation.cc:769] Sending connection context
0806 17:05:05.487712 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 17:05:05.487890 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260806 17:05:05.488412 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.474503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58472 (local address 127.9.111.125:40811)
0806 17:05:05.475052 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:05.475067 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:05.475096 (+ 29us) server_negotiation.cc:407] Connection header received
0806 17:05:05.475149 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:05.475154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:05.475404 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:05.475591 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:05.476050 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.476971 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.477794 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.478409 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.478911 (+ 502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.483735 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:05.483743 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:05.483758 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:05.483785 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:05.485928 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.486495 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.486500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.486502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.486602 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.486921 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.486927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.486931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.487306 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:05.487491 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:05.487821 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 17:05:05.488076 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":122,"threads_started":1}
I20260806 17:05:05.490443 23503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } has no permanent_uuid. Determining permanent_uuid...
I20260806 17:05:05.491343 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.490950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:42953 (local address 127.0.0.1:44430)
0806 17:05:05.491071 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.491263 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 17:05:05.492141 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.491851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:42953 (local address 127.0.0.1:44432)
0806 17:05:05.492001 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.492073 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260806 17:05:05.492322 23503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.124:42953: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
Aug 06 17:05:05 dist-test-slave-69rg krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035905, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.125@KRBTEST.COM for kudu/127.9.111.125@KRBTEST.COM
I20260806 17:05:05.501968 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.487929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:40811 (local address 127.0.0.1:58478)
0806 17:05:05.488197 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.488212 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:05.488297 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:05.489010 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:05.489015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:05.489214 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:05.489362 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:05.489374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.490572 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.490578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:05.491488 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:05.491504 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.492399 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.492405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:05.492516 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.493016 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:05.493040 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:05.497784 (+ 4744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:05.500136 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:05.500139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:05.500141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:05.500360 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:05.500779 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:05.500782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:05.500783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:05.500872 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:05.501388 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:05.501391 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:05.501484 (+ 93us) client_negotiation.cc:769] Sending connection context
0806 17:05:05.501580 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 17:05:05.501761 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260806 17:05:05.502077 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.488673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58478 (local address 127.9.111.125:40811)
0806 17:05:05.488792 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:05.488796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:05.488807 (+ 11us) server_negotiation.cc:407] Connection header received
0806 17:05:05.488840 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:05.488843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:05.488883 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:05.488963 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:05.489500 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.490438 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.491634 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.492106 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.492271 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.498203 (+ 5932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:05.498207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:05.498209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:05.498240 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:05.499995 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.500535 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.500563 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.500564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.500658 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.500983 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.500986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.500987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.501278 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:05.501359 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:05.501757 (+ 398us) server_negotiation.cc:299] Negotiation successful
0806 17:05:05.501931 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 17:05:05.502956 23579 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } has no permanent_uuid. Determining permanent_uuid...
I20260806 17:05:05.503713 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.503448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:42953 (local address 127.0.0.1:44448)
0806 17:05:05.503568 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.503657 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 17:05:05.504299 23527 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.124.keytab.
Entry for principal kudu/127.9.111.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.124.keytab.
I20260806 17:05:05.505205 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.504993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:42953 (local address 127.0.0.1:44458)
0806 17:05:05.505101 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.505148 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 17:05:05.505386 23579 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.124:42953: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
I20260806 17:05:05.505865 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.124:42953
--webserver_interface=127.9.111.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.124.keytab
--principal=kudu/127.9.111.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:42607,127.9.111.125:40811,127.9.111.124:42953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:05:05.523626 23503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } attempt: 1
I20260806 17:05:05.524712 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.524346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:42953 (local address 127.0.0.1:44462)
0806 17:05:05.524532 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.524609 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 17:05:05.525754 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.525494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:42953 (local address 127.0.0.1:44466)
0806 17:05:05.525617 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.525678 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 17:05:05.525970 23503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.124:42953: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
I20260806 17:05:05.554689 23579 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } attempt: 1
I20260806 17:05:05.555686 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.555351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:42953 (local address 127.0.0.1:44478)
0806 17:05:05.555536 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.555602 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260806 17:05:05.556465 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.556237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:42953 (local address 127.0.0.1:44482)
0806 17:05:05.556351 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.556405 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 17:05:05.556600 23579 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.124:42953: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
I20260806 17:05:05.601174 23503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } attempt: 2
I20260806 17:05:05.602173 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.601786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:42953 (local address 127.0.0.1:44490)
0806 17:05:05.602016 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.602084 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260806 17:05:05.603042 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.602823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:42953 (local address 127.0.0.1:44500)
0806 17:05:05.602931 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.602980 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 17:05:05.603261 23503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.124:42953: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
I20260806 17:05:05.624800 23579 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } attempt: 2
I20260806 17:05:05.625795 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.625458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:42953 (local address 127.0.0.1:44514)
0806 17:05:05.625623 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.625710 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 17:05:05.626665 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.626417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:42953 (local address 127.0.0.1:44520)
0806 17:05:05.626537 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.626595 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 17:05:05.626837 23579 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.124:42953: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
W20260806 17:05:05.632324 23587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:05.632647 23587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:05.632691 23587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:05.637167 23587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:05:05.637255 23587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:05.637275 23587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:05.637293 23587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:05:05.637310 23587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:05:05.643098 23587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:42607,127.9.111.125:40811,127.9.111.124:42953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.124.keytab
--principal=kudu/127.9.111.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.124:42953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.111.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:05.644301 23587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:05.645474 23587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:05.655444 23592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:05.655393 23595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:05.656172 23593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:05.656203 23587 server_base.cc:1034] running on GCE node
I20260806 17:05:05.656926 23587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:05.658273 23587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:05.659482 23587 hybrid_clock.cc:648] HybridClock initialized: now 1786035905659466 us; error 33 us; skew 500 ppm
Aug 06 17:05:05 dist-test-slave-69rg krb5kdc[23415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035905, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:05.662781 23587 init.cc:377] Logged in from keytab as kudu/127.9.111.124@KRBTEST.COM (short username kudu)
I20260806 17:05:05.663828 23587 webserver.cc:460] Webserver started at http://127.9.111.124:45757/ using document root <none> and password file <none>
I20260806 17:05:05.664387 23587 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:05.664458 23587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:05.664642 23587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:05:05.667704 23587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-2/data/instance:
uuid: "bf142594e46f43dd91fb6640717cd1fb"
format_stamp: "Formatted at 2026-08-06 17:05:05 on dist-test-slave-69rg"
server_key: "9a5a53212aecf2e17f36bbadee9cb5b5"
server_key_iv: "81b7eb3a0fb4c270499ce8d5228a3c59"
server_key_version: "encryptionkey@0"
I20260806 17:05:05.668386 23587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-2/wal/instance:
uuid: "bf142594e46f43dd91fb6640717cd1fb"
format_stamp: "Formatted at 2026-08-06 17:05:05 on dist-test-slave-69rg"
server_key: "9a5a53212aecf2e17f36bbadee9cb5b5"
server_key_iv: "81b7eb3a0fb4c270499ce8d5228a3c59"
server_key_version: "encryptionkey@0"
I20260806 17:05:05.672832 23587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 17:05:05.675932 23602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:05.677596 23587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 17:05:05.677794 23587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-2/wal
uuid: "bf142594e46f43dd91fb6640717cd1fb"
format_stamp: "Formatted at 2026-08-06 17:05:05 on dist-test-slave-69rg"
server_key: "9a5a53212aecf2e17f36bbadee9cb5b5"
server_key_iv: "81b7eb3a0fb4c270499ce8d5228a3c59"
server_key_version: "encryptionkey@0"
I20260806 17:05:05.678026 23587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:05.695508 23587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:05.699064 23587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:05.699311 23587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:05.702431 23503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } attempt: 3
I20260806 17:05:05.703452 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.703015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:42953 (local address 127.0.0.1:44530)
0806 17:05:05.703262 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.703341 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 17:05:05.704416 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.704101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:42953 (local address 127.0.0.1:44536)
0806 17:05:05.704250 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.704326 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 17:05:05.704597 23503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.124:42953: Network error: Client connection negotiation failed: client connection to 127.9.111.124:42953: connect: Connection refused (error 111)
I20260806 17:05:05.708365 23587 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.124:42953
I20260806 17:05:05.708393 23657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.124:42953 every 8 connection(s)
I20260806 17:05:05.709475 23587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-2/data/info.pb
I20260806 17:05:05.712807 23658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:05.714763 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 23587
I20260806 17:05:05.714905 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786035732873553-9661-0/minicluster-data/master-2/wal/instance
I20260806 17:05:05.715201 9661 external_mini_cluster.cc:1428] Setting key b070790b00c6d8cb551c9187c4b69f9f
I20260806 17:05:05.718015 23579 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } attempt: 3
I20260806 17:05:05.717547 23658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 17:05:05 dist-test-slave-69rg krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035905, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.125@KRBTEST.COM for kudu/127.9.111.124@KRBTEST.COM
Aug 06 17:05:05 dist-test-slave-69rg krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035905, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.124@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
Aug 06 17:05:05 dist-test-slave-69rg krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.124@KRBTEST.COM
I20260806 17:05:05.731833 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.718623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:42953 (local address 127.0.0.1:44548)
0806 17:05:05.718847 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.718867 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:05.718988 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:05.719620 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:05.719629 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:05.719820 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:05.719913 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:05.719919 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.721769 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.721773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:05.722650 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:05.722665 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.723714 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.723719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:05.723820 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.724372 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:05.724391 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:05.726186 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:05.729071 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:05.729075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:05.729078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:05.729312 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:05.729741 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:05.729746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:05.729751 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:05.729893 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:05.731165 (+ 1272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:05.731169 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:05.731306 (+ 137us) client_negotiation.cc:769] Sending connection context
0806 17:05:05.731428 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 17:05:05.731610 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260806 17:05:05.732873 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.720919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46720 (local address 127.9.111.126:42607)
0806 17:05:05.721080 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:05.721085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:05.721102 (+ 17us) server_negotiation.cc:407] Connection header received
0806 17:05:05.721146 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:05.721149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:05.721192 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:05.721282 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:05.722264 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.722857 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.724067 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.724388 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.724518 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.727287 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:05.727293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:05.727300 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:05.727347 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:05.729539 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.730258 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.730263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.730264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.730366 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.730939 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.730942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.730945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.731250 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:05.731364 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:05.732252 (+ 888us) server_negotiation.cc:299] Negotiation successful
0806 17:05:05.732514 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 17:05:05.734146 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.716802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44540 (local address 127.9.111.124:42953)
0806 17:05:05.717328 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:05.717345 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:05.717385 (+ 40us) server_negotiation.cc:407] Connection header received
0806 17:05:05.718208 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:05.718237 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:05.718711 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:05.719088 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:05.720113 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.721518 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.722853 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.723574 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.723723 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.727974 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:05.727978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:05.727979 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:05.728023 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:05.730505 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.731068 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.731071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.731073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.731153 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.731493 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.731497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.731498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.731700 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:05.731872 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:05.732679 (+ 807us) server_negotiation.cc:299] Negotiation successful
0806 17:05:05.732937 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":376,"thread_start_us":174,"threads_started":1}
I20260806 17:05:05.734138 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.720134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:42607 (local address 127.0.0.1:46720)
0806 17:05:05.720535 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.720587 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:05.720711 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:05.721386 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:05.721392 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:05.721968 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:05.722096 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:05.722102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.722978 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.722981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:05.723932 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:05.723941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.724584 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.724591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:05.724740 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.725280 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:05.725298 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:05.727032 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:05.729684 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:05.729692 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:05.729697 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:05.730127 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:05.730722 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:05.730725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:05.730727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:05.730814 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:05.731375 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:05.731384 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:05.731557 (+ 173us) client_negotiation.cc:769] Sending connection context
0806 17:05:05.733090 (+ 1533us) client_negotiation.cc:240] Negotiation successful
0806 17:05:05.733402 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":110,"thread_start_us":127,"threads_started":1}
I20260806 17:05:05.734138 23662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.718664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.9.111.124:42953)
0806 17:05:05.719339 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:05.719349 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:05.719363 (+ 14us) server_negotiation.cc:407] Connection header received
0806 17:05:05.719404 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:05.719407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:05.719452 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:05.719561 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:05.720066 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.721549 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.722800 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.723570 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.723730 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.726375 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:05.726382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:05.726394 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:05.726418 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:05.728932 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.729445 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.729452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.729464 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.729606 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.730007 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.730009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.730011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.730986 (+ 975us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:05.731186 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:05.731576 (+ 390us) server_negotiation.cc:299] Negotiation successful
0806 17:05:05.733390 (+ 1814us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":843,"server-negotiator.queue_time_us":597,"spinlock_wait_cycles":9088,"thread_start_us":194,"threads_started":1}
I20260806 17:05:05.735935 23658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } has no permanent_uuid. Determining permanent_uuid...
I20260806 17:05:05.739951 23579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a: Bootstrap starting.
Aug 06 17:05:05 dist-test-slave-69rg krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:05:05.743255 23579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:05.744234 23579 log.cc:824] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:05.747247 23579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a: No bootstrap required, opened a new log
Aug 06 17:05:05 dist-test-slave-69rg krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035905, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.124@KRBTEST.COM for kudu/127.9.111.125@KRBTEST.COM
I20260806 17:05:05.748466 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.737079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46732 (local address 127.9.111.126:42607)
0806 17:05:05.737244 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:05.737249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:05.737268 (+ 19us) server_negotiation.cc:407] Connection header received
0806 17:05:05.737318 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:05.737324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:05.737386 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:05.737515 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:05.738339 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.739088 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.740110 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.740429 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.740549 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.743931 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:05.743935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:05.743938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:05.743975 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:05.746295 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.747000 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.747004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.747005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.747114 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.747497 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.747500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.747502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.747751 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:05.747998 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:05.748147 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 17:05:05.748301 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 17:05:05.752257 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.736532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:40811 (local address 127.0.0.1:58490)
0806 17:05:05.736726 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.736742 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:05.736826 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:05.742550 (+ 5724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:05.742556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:05.742808 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:05.742950 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:05.742958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.743853 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.743856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:05.744945 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:05.744961 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.745638 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.745642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:05.745764 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.746289 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:05.746309 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:05.748056 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:05.750222 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:05.750227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:05.750232 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:05.750509 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:05.750870 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:05.750873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:05.750875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:05.750980 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:05.751597 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:05.751600 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:05.751724 (+ 124us) client_negotiation.cc:769] Sending connection context
0806 17:05:05.751872 (+ 148us) client_negotiation.cc:240] Negotiation successful
0806 17:05:05.752041 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260806 17:05:05.752142 23579 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } } peers { permanent_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } } peers { permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } }
I20260806 17:05:05.752300 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.742020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58490 (local address 127.9.111.125:40811)
0806 17:05:05.742185 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:05.742190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:05.742208 (+ 18us) server_negotiation.cc:407] Connection header received
0806 17:05:05.742255 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:05.742258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:05.742308 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:05.742409 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:05.743099 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.743706 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.745105 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.745511 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.745619 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.748228 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:05.748232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:05.748235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:05.748268 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:05.750085 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.750639 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.750643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.750645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.750747 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.751109 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.751115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.751119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.751442 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:05.751563 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:05.751957 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 17:05:05.752076 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 17:05:05.752480 23579 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:05.752985 23579 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf0ee50c23da4c13a30e08b4d437df1a, State: Initialized, Role: FOLLOWER
I20260806 17:05:05.753607 23658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } has no permanent_uuid. Determining permanent_uuid...
I20260806 17:05:05.753605 23579 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } } peers { permanent_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } } peers { permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } }
I20260806 17:05:05.754197 23666 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } } peers { permanent_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } } peers { permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } } }
I20260806 17:05:05.754356 23666 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:05:05.755144 23579 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [sys.catalog]: configured and running, proceeding with master startup.
Aug 06 17:05:05 dist-test-slave-69rg krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.125@KRBTEST.COM
Aug 06 17:05:05 dist-test-slave-69rg krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035905, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.124@KRBTEST.COM for kudu/127.9.111.124@KRBTEST.COM
W20260806 17:05:05.762692 23678 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260806 17:05:05.762770 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.750136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58502 (local address 127.9.111.125:40811)
0806 17:05:05.750543 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:05.750555 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:05.750571 (+ 16us) server_negotiation.cc:407] Connection header received
0806 17:05:05.750613 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:05.750616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:05.750661 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:05.750767 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:05.751584 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.752302 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.753563 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.753995 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.754519 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.757254 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:05.757258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:05.757260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:05.757294 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:05.759419 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.759999 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.760006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.760008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.760729 (+ 721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.761204 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.761207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.761209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.761448 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:05.761650 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:05.762406 (+ 756us) server_negotiation.cc:299] Negotiation successful
0806 17:05:05.762537 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":145,"threads_started":1}
W20260806 17:05:05.762791 23678 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:05:05.764333 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.754209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:42953 (local address 127.0.0.1:44554)
0806 17:05:05.754805 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.754823 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:05.754910 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:05.755506 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:05.755513 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:05.755718 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:05.755819 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:05.755826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.756921 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.756926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:05.757778 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:05.757791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.758472 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.758475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:05.758545 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.758904 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:05.758918 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:05.760434 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:05.762630 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:05.762633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:05.762636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:05.762903 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:05.763171 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:05.763174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:05.763175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:05.763252 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:05.763796 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:05.763800 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:05.763905 (+ 105us) client_negotiation.cc:769] Sending connection context
0806 17:05:05.764019 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 17:05:05.764161 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260806 17:05:05.759400 23510 master_runner.cc:186] Error getting master registration for 127.9.111.126:42607: OK, instance_id { permanent_uuid: "284269db8738422b81edcce2505564c6" instance_seqno: 1786035905224285 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 17:05:05.764703 23510 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 17:05:05.765141 23662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.755054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.9.111.124:42953)
0806 17:05:05.755211 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:05.755215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:05.755230 (+ 15us) server_negotiation.cc:407] Connection header received
0806 17:05:05.755322 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:05.755325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:05.755369 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:05.755452 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:05.755955 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.756763 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.757945 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.758256 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.758357 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.760571 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:05.760574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:05.760578 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:05.760601 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:05.762510 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.763009 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.763013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.763015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.763088 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.763342 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.763345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.763347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.763657 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:05.763778 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:05.764887 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0806 17:05:05.765016 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 17:05:05.766709 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:05:05.768882 23658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb: Bootstrap starting.
I20260806 17:05:05.772370 23658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:05.773412 23658 log.cc:824] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:05.780319 23658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb: No bootstrap required, opened a new log
I20260806 17:05:05.782265 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.769579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58516 (local address 127.9.111.125:40811)
0806 17:05:05.769863 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:05.769868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:05.770650 (+ 782us) server_negotiation.cc:407] Connection header received
0806 17:05:05.770889 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:05.770893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:05.770945 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:05.771004 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:05.772100 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.773248 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.774818 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.775196 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.775281 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.778180 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:05.778185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:05.778189 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:05.778221 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:05.780011 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.780619 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.780624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.780626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.780731 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.781030 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.781033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.781035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.781274 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:05.781413 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:05.781884 (+ 471us) server_negotiation.cc:299] Negotiation successful
0806 17:05:05.782051 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260806 17:05:05.785306 23662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.770732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44562 (local address 127.9.111.124:42953)
0806 17:05:05.770856 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:05.770861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:05.774733 (+ 3872us) server_negotiation.cc:407] Connection header received
0806 17:05:05.774920 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:05.774923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:05.774974 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:05.775028 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:05.776118 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.776884 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.778140 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.778484 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.778602 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.781092 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:05.781096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:05.781099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:05.781128 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:05.782926 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.783620 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.783624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.783626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.783740 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.784070 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.784073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.784075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.784306 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:05.784431 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:05.785015 (+ 584us) server_negotiation.cc:299] Negotiation successful
0806 17:05:05.785141 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 17:05:05.785768 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.769969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.9.111.126:42607)
0806 17:05:05.770095 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:05.770100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:05.771149 (+ 1049us) server_negotiation.cc:407] Connection header received
0806 17:05:05.771367 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:05.771371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:05.771427 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:05.771496 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:05.772616 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.773464 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.774754 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.775128 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.775217 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.777541 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:05.777545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:05.777548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:05.777582 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:05.780897 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.784084 (+ 3187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.784089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.784091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.784219 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.784598 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.784604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.784606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.784881 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:05.785023 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:05.785423 (+ 400us) server_negotiation.cc:299] Negotiation successful
0806 17:05:05.785571 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 17:05:05.787451 23658 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } } peers { permanent_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } } peers { permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } }
I20260806 17:05:05.787705 23658 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:05.787747 23658 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf142594e46f43dd91fb6640717cd1fb, State: Initialized, Role: FOLLOWER
I20260806 17:05:05.788156 23658 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } } peers { permanent_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } } peers { permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } }
I20260806 17:05:05.788671 23688 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } } peers { permanent_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } } peers { permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } } }
I20260806 17:05:05.788864 23688 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:05:05.789676 23658 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb [sys.catalog]: configured and running, proceeding with master startup.
W20260806 17:05:05.796816 23700 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:05:05.796934 23700 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:05:05.794523 23587 master_runner.cc:186] Error getting master registration for 127.9.111.126:42607: OK, instance_id { permanent_uuid: "284269db8738422b81edcce2505564c6" instance_seqno: 1786035905224285 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 17:05:05.798130 23587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 17:05:05.868790 23503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } attempt: 4
Aug 06 17:05:05 dist-test-slave-69rg krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035905, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for kudu/127.9.111.124@KRBTEST.COM
I20260806 17:05:05.879415 23662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.869391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44564 (local address 127.9.111.124:42953)
0806 17:05:05.869519 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:05.869523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:05.869594 (+ 71us) server_negotiation.cc:407] Connection header received
0806 17:05:05.869718 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:05.869729 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:05.869779 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:05.869899 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:05.870547 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.871363 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.872515 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.872873 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.872983 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.875811 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:05.875815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:05.875818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:05.875849 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:05.877559 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.878087 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.878090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.878092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.878189 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:05.878487 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:05.878490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:05.878492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:05.878714 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:05.878818 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:05.879125 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 17:05:05.879250 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 17:05:05.879419 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:05.869319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:42953 (local address 127.0.0.1:44564)
0806 17:05:05.869517 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:05.869532 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:05.869624 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:05.870044 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:05.870047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:05.870277 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:05.870397 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:05.870405 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.871491 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.871495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:05.872366 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:05.872379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:05.873003 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:05.873008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:05.873106 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:05.873762 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:05.873779 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:05.875590 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:05.877683 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:05.877686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:05.877689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:05.877992 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:05.878290 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:05.878293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:05.878296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:05.878393 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:05.878828 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:05.878831 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:05.878948 (+ 117us) client_negotiation.cc:769] Sending connection context
0806 17:05:05.879065 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 17:05:05.879215 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260806 17:05:05.883386 23503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6: Bootstrap starting.
I20260806 17:05:05.886390 23503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:05.887348 23503 log.cc:824] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:05.889770 23503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6: No bootstrap required, opened a new log
I20260806 17:05:05.894615 23503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } } peers { permanent_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } } peers { permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } }
I20260806 17:05:05.894908 23503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:05.895021 23503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 284269db8738422b81edcce2505564c6, State: Initialized, Role: FOLLOWER
I20260806 17:05:05.895565 23503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } } peers { permanent_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } } peers { permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } }
I20260806 17:05:05.896250 23703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } } peers { permanent_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } } peers { permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } } }
I20260806 17:05:05.896484 23703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:05:05.897133 23503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 17:05:05.905668 23715 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:05:05.905892 23715 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:05:05.906657 23429 master_runner.cc:263] No leader master found from master 284269db8738422b81edcce2505564c6
I20260806 17:05:05.906769 23429 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 17:05:05.950883 23608 tablet.cc:2369] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 17:05:05.951680 23531 tablet.cc:2369] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 17:05:05.971238 23451 tablet.cc:2369] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:05:06.034492 23666 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:05:06.034801 23666 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } } peers { permanent_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } } peers { permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } }
I20260806 17:05:06.035856 23666 leader_election.cc:288] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 284269db8738422b81edcce2505564c6 (127.9.111.126:42607), bf142594e46f43dd91fb6640717cd1fb (127.9.111.124:42953)
I20260806 17:05:06.036521 23475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284269db8738422b81edcce2505564c6" is_pre_election: true
I20260806 17:05:06.036516 23631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf142594e46f43dd91fb6640717cd1fb" is_pre_election: true
I20260806 17:05:06.036870 23475 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf0ee50c23da4c13a30e08b4d437df1a in term 0.
I20260806 17:05:06.036870 23631 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf0ee50c23da4c13a30e08b4d437df1a in term 0.
I20260806 17:05:06.037355 23527 leader_election.cc:302] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 284269db8738422b81edcce2505564c6, cf0ee50c23da4c13a30e08b4d437df1a; no voters:
I20260806 17:05:06.037637 23666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:05:06.037742 23666 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:05:06.037797 23666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:06.038918 23666 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } } peers { permanent_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } } peers { permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } }
I20260806 17:05:06.039260 23666 leader_election.cc:288] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [CANDIDATE]: Term 1 election: Requested vote from peers 284269db8738422b81edcce2505564c6 (127.9.111.126:42607), bf142594e46f43dd91fb6640717cd1fb (127.9.111.124:42953)
I20260806 17:05:06.039713 23475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284269db8738422b81edcce2505564c6"
I20260806 17:05:06.039713 23631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf142594e46f43dd91fb6640717cd1fb"
I20260806 17:05:06.039855 23475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:06.039877 23631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:06.041211 23631 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf0ee50c23da4c13a30e08b4d437df1a in term 1.
I20260806 17:05:06.041211 23475 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf0ee50c23da4c13a30e08b4d437df1a in term 1.
I20260806 17:05:06.041759 23527 leader_election.cc:302] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf142594e46f43dd91fb6640717cd1fb, cf0ee50c23da4c13a30e08b4d437df1a; no voters:
I20260806 17:05:06.042047 23666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:05:06.042297 23666 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [term 1 LEADER]: Becoming Leader. State: Replica: cf0ee50c23da4c13a30e08b4d437df1a, State: Running, Role: LEADER
I20260806 17:05:06.042695 23666 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } } peers { permanent_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } } peers { permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } }
I20260806 17:05:06.044759 23717 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf0ee50c23da4c13a30e08b4d437df1a. Latest consensus state: current_term: 1 leader_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } } peers { permanent_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } } peers { permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } } }
I20260806 17:05:06.044924 23717 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [sys.catalog]: This master's current role is: LEADER
I20260806 17:05:06.045325 23719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:05:06.046265 23719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:05:06.050961 23631 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb [term 1 FOLLOWER]: Refusing update from remote peer cf0ee50c23da4c13a30e08b4d437df1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:05:06.050961 23475 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6 [term 1 FOLLOWER]: Refusing update from remote peer cf0ee50c23da4c13a30e08b4d437df1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:05:06.051646 23666 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:05:06.051940 23717 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:05:06.056416 23688 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf0ee50c23da4c13a30e08b4d437df1a. Latest consensus state: current_term: 1 leader_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } } peers { permanent_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } } peers { permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } } }
I20260806 17:05:06.056620 23688 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:05:06.056538 23703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf0ee50c23da4c13a30e08b4d437df1a. Latest consensus state: current_term: 1 leader_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } } peers { permanent_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } } peers { permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } } }
I20260806 17:05:06.056691 23703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:05:06.059881 23666 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } } peers { permanent_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } } peers { permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } } }
I20260806 17:05:06.060041 23666 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [sys.catalog]: This master's current role is: LEADER
I20260806 17:05:06.061302 23688 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } } peers { permanent_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } } peers { permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } } }
I20260806 17:05:06.061553 23688 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:05:06.061776 23666 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } } peers { permanent_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } } peers { permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } } }
I20260806 17:05:06.061928 23666 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a [sys.catalog]: This master's current role is: LEADER
I20260806 17:05:06.062462 23722 mvcc.cc:205] Tried to move back new op lower bound from 7315603071177486336 to 7315603071153745920. Current Snapshot: MvccSnapshot[applied={T|T < 7315603071177486336}]
I20260806 17:05:06.063361 23703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284269db8738422b81edcce2505564c6" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42607 } } peers { permanent_uuid: "cf0ee50c23da4c13a30e08b4d437df1a" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 40811 } } peers { permanent_uuid: "bf142594e46f43dd91fb6640717cd1fb" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 42953 } } }
I20260806 17:05:06.063506 23703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:05:06.066939 23719 catalog_manager.cc:1347] Generated new cluster ID: 6d513cb84868478882982d5543e05996
I20260806 17:05:06.067009 23719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:05:06.069250 23721 mvcc.cc:205] Tried to move back new op lower bound from 7315603071177486336 to 7315603071153745920. Current Snapshot: MvccSnapshot[applied={T|T < 7315603071177486336}]
I20260806 17:05:06.098032 23719 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:05:06.099115 23719 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:05:06.117445 23719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf0ee50c23da4c13a30e08b4d437df1a: Generated new TSK 0
I20260806 17:05:06.118608 23719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:05:06.798637 23700 catalog_manager.cc:1259] Loaded cluster ID: 6d513cb84868478882982d5543e05996
I20260806 17:05:06.798722 23700 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb: loading cluster ID for follower catalog manager: success
I20260806 17:05:06.800464 23700 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb: acquiring CA information for follower catalog manager: success
I20260806 17:05:06.801662 23700 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf142594e46f43dd91fb6640717cd1fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 17:05:06.907387 23715 catalog_manager.cc:1259] Loaded cluster ID: 6d513cb84868478882982d5543e05996
I20260806 17:05:06.907527 23715 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6: loading cluster ID for follower catalog manager: success
I20260806 17:05:06.909564 23715 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6: acquiring CA information for follower catalog manager: success
I20260806 17:05:06.910984 23715 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 284269db8738422b81edcce2505564c6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 17:05:07.198988 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 23429
I20260806 17:05:07.206422 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 23510
I20260806 17:05:07.213550 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 23587
2026-08-06T17:05:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3400 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:05:07 dist-test-slave-69rg krb5kdc[23736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:05:07 dist-test-slave-69rg krb5kdc[23736](info): set up 2 sockets
Aug 06 17:05:07 dist-test-slave-69rg krb5kdc[23736](info): commencing operation
krb5kdc: starting...
W20260806 17:05:08.289454 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:05:08 dist-test-slave-69rg krb5kdc[23736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:05:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:05:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:05:08.432955 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:34959
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:34959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:05:08.547744 23749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:08.548110 23749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:08.548187 23749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:08.552441 23749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:05:08.552574 23749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:08.552611 23749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:08.552654 23749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:05:08.552686 23749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:05:08.558447 23749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:34959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:34959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:08.559671 23749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:08.560773 23749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:08.570032 23755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:08.570010 23756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:08.570132 23749 server_base.cc:1034] running on GCE node
W20260806 17:05:08.570421 23759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:08.571020 23749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:08.572157 23749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:08.573350 23749 hybrid_clock.cc:648] HybridClock initialized: now 1786035908573332 us; error 31 us; skew 500 ppm
Aug 06 17:05:08 dist-test-slave-69rg krb5kdc[23736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035908, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:08.576174 23749 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:05:08.577131 23749 webserver.cc:460] Webserver started at http://127.9.111.126:39719/ using document root <none> and password file <none>
I20260806 17:05:08.577705 23749 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:08.577862 23749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:08.578114 23749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:05:08.580289 23749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "53cbab699f9b4b80a733015071130194"
format_stamp: "Formatted at 2026-08-06 17:05:08 on dist-test-slave-69rg"
server_key: "4848dafa3ca75f0ce07eedea7bcbb3b1"
server_key_iv: "710d1869adb5a1f3bed40ff4f36a11a6"
server_key_version: "encryptionkey@0"
I20260806 17:05:08.580884 23749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "53cbab699f9b4b80a733015071130194"
format_stamp: "Formatted at 2026-08-06 17:05:08 on dist-test-slave-69rg"
server_key: "4848dafa3ca75f0ce07eedea7bcbb3b1"
server_key_iv: "710d1869adb5a1f3bed40ff4f36a11a6"
server_key_version: "encryptionkey@0"
I20260806 17:05:08.585124 23749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 17:05:08.587986 23766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:08.589625 23749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 17:05:08.589852 23749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "53cbab699f9b4b80a733015071130194"
format_stamp: "Formatted at 2026-08-06 17:05:08 on dist-test-slave-69rg"
server_key: "4848dafa3ca75f0ce07eedea7bcbb3b1"
server_key_iv: "710d1869adb5a1f3bed40ff4f36a11a6"
server_key_version: "encryptionkey@0"
I20260806 17:05:08.590011 23749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:08.603529 23749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:08.607084 23749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:08.607335 23749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:08.615615 23749 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:34959
I20260806 17:05:08.615634 23820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:34959 every 8 connection(s)
I20260806 17:05:08.616693 23749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:05:08.620051 23821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:08.621181 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 23749
I20260806 17:05:08.621296 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:05:08.621703 9661 external_mini_cluster.cc:1428] Setting key 6262f0d0168d7526ca54c7c051e1999b
I20260806 17:05:08.627179 23821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194: Bootstrap starting.
I20260806 17:05:08.629969 23821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:08.630960 23821 log.cc:824] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194: Log is configured to *not* fsync() on all Append() calls
Aug 06 17:05:08 dist-test-slave-69rg krb5kdc[23736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:05:08.633286 23821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194: No bootstrap required, opened a new log
I20260806 17:05:08.637064 23821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cbab699f9b4b80a733015071130194" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34959 } }
I20260806 17:05:08.637315 23821 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:08.637351 23821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53cbab699f9b4b80a733015071130194, State: Initialized, Role: FOLLOWER
I20260806 17:05:08.637478 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:08.623290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42932 (local address 127.9.111.126:34959)
0806 17:05:08.623811 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:08.623827 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:08.623864 (+ 37us) server_negotiation.cc:407] Connection header received
0806 17:05:08.624519 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:08.624535 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:08.624871 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:08.625260 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:08.626125 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:08.627110 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:08.628060 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:08.628628 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:08.628814 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:08.632051 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:08.632058 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:08.632073 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:08.632114 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:08.634167 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:08.634695 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:08.634701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:08.634706 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:08.634834 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:08.635161 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:08.635166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:08.635168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:08.635611 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:08.635840 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:08.636181 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 17:05:08.636521 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":185,"threads_started":1}
I20260806 17:05:08.637943 23821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cbab699f9b4b80a733015071130194" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34959 } }
I20260806 17:05:08.638127 23821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:05:08.638221 23821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:05:08.638345 23821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:08.639642 23821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cbab699f9b4b80a733015071130194" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34959 } }
I20260806 17:05:08.640030 23821 leader_election.cc:302] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53cbab699f9b4b80a733015071130194; no voters:
I20260806 17:05:08.640400 23821 leader_election.cc:288] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:05:08.640527 23826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:05:08.640823 23826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194 [term 1 LEADER]: Becoming Leader. State: Replica: 53cbab699f9b4b80a733015071130194, State: Running, Role: LEADER
I20260806 17:05:08.641101 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:05:08.641170 23826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cbab699f9b4b80a733015071130194" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34959 } }
I20260806 17:05:08.641860 23821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:05:08.642985 23828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53cbab699f9b4b80a733015071130194. Latest consensus state: current_term: 1 leader_uuid: "53cbab699f9b4b80a733015071130194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cbab699f9b4b80a733015071130194" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34959 } } }
I20260806 17:05:08.643142 23828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194 [sys.catalog]: This master's current role is: LEADER
I20260806 17:05:08.643549 23827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53cbab699f9b4b80a733015071130194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cbab699f9b4b80a733015071130194" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34959 } } }
I20260806 17:05:08.643603 23835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:05:08.643664 23827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194 [sys.catalog]: This master's current role is: LEADER
I20260806 17:05:08.647030 23835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:05:08.652823 23835 catalog_manager.cc:1347] Generated new cluster ID: 035413de9de14f48a22f3b3d6f56bbc1
I20260806 17:05:08.652923 23835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:05:08.676891 23835 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:05:08.678157 23835 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:05:08.686808 23835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53cbab699f9b4b80a733015071130194: Generated new TSK 0
I20260806 17:05:08.687649 23835 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:05:08.697595 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 23749
2026-08-06T17:05:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1482 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:05:08 dist-test-slave-69rg krb5kdc[23848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:05:08 dist-test-slave-69rg krb5kdc[23848](info): set up 2 sockets
Aug 06 17:05:08 dist-test-slave-69rg krb5kdc[23848](info): commencing operation
krb5kdc: starting...
W20260806 17:05:09.769343 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:05:09 dist-test-slave-69rg krb5kdc[23848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:05:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:05:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:05:09.908717 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:37601
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:37601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:05:10.025648 23862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:10.026055 23862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:10.026166 23862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:10.030462 23862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:05:10.030576 23862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:10.030619 23862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:10.030659 23862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:05:10.030694 23862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:05:10.036516 23862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:37601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:37601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:10.037778 23862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:10.038949 23862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:10.047787 23869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:10.047835 23868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:10.047845 23871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:10.048069 23862 server_base.cc:1034] running on GCE node
I20260806 17:05:10.048799 23862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:10.050061 23862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:10.051282 23862 hybrid_clock.cc:648] HybridClock initialized: now 1786035910051236 us; error 60 us; skew 500 ppm
Aug 06 17:05:10 dist-test-slave-69rg krb5kdc[23848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035910, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:10.065851 23862 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:05:10.067417 23862 webserver.cc:460] Webserver started at http://127.9.111.126:33425/ using document root <none> and password file <none>
I20260806 17:05:10.068307 23862 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:10.068411 23862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:10.068646 23862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:05:10.071055 23862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "d28bb840247d47e3a039d2885030f6ec"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "555b15fbde0b497b35a76cb59ed6b938"
server_key_iv: "5a8e6ca5ef3df8c8e076fc8039f15ba3"
server_key_version: "encryptionkey@0"
I20260806 17:05:10.071780 23862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "d28bb840247d47e3a039d2885030f6ec"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "555b15fbde0b497b35a76cb59ed6b938"
server_key_iv: "5a8e6ca5ef3df8c8e076fc8039f15ba3"
server_key_version: "encryptionkey@0"
I20260806 17:05:10.075941 23862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 17:05:10.078933 23878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:10.080238 23862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:05:10.080435 23862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "d28bb840247d47e3a039d2885030f6ec"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "555b15fbde0b497b35a76cb59ed6b938"
server_key_iv: "5a8e6ca5ef3df8c8e076fc8039f15ba3"
server_key_version: "encryptionkey@0"
I20260806 17:05:10.080572 23862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:10.105361 23862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:10.108912 23862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:10.109169 23862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:10.116947 23862 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:37601
I20260806 17:05:10.117260 23933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:37601 every 8 connection(s)
I20260806 17:05:10.118021 23862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:05:10.120888 23935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:10.126933 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 23862
I20260806 17:05:10.127080 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:05:10.127420 9661 external_mini_cluster.cc:1428] Setting key 7f713fd1f42163511f8d469fb4fc9312
I20260806 17:05:10.128394 23935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec: Bootstrap starting.
I20260806 17:05:10.131421 23935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:10.132423 23935 log.cc:824] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:10.134631 23935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec: No bootstrap required, opened a new log
Aug 06 17:05:10 dist-test-slave-69rg krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:05:10.138643 23935 raft_consensus.cc:348] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28bb840247d47e3a039d2885030f6ec" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37601 } }
I20260806 17:05:10.138918 23935 raft_consensus.cc:374] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:10.139003 23935 raft_consensus.cc:729] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d28bb840247d47e3a039d2885030f6ec, State: Initialized, Role: FOLLOWER
I20260806 17:05:10.139523 23935 consensus_queue.cc:260] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28bb840247d47e3a039d2885030f6ec" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37601 } }
I20260806 17:05:10.139700 23935 raft_consensus.cc:388] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:05:10.139803 23935 raft_consensus.cc:482] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:05:10.139932 23935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:10.140983 23935 raft_consensus.cc:504] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28bb840247d47e3a039d2885030f6ec" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37601 } }
I20260806 17:05:10.141458 23935 leader_election.cc:302] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d28bb840247d47e3a039d2885030f6ec; no voters:
I20260806 17:05:10.141956 23935 leader_election.cc:288] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:05:10.142040 23940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:05:10.142280 23940 raft_consensus.cc:686] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 1 LEADER]: Becoming Leader. State: Replica: d28bb840247d47e3a039d2885030f6ec, State: Running, Role: LEADER
I20260806 17:05:10.142638 23940 consensus_queue.cc:237] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28bb840247d47e3a039d2885030f6ec" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37601 } }
I20260806 17:05:10.143376 23935 sys_catalog.cc:564] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:05:10.142282 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:10.129043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.9.111.126:37601)
0806 17:05:10.129511 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:10.129528 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:10.129565 (+ 37us) server_negotiation.cc:407] Connection header received
0806 17:05:10.130329 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:10.130353 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:10.130711 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:10.131001 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:10.132083 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.133194 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.134112 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.134588 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.134762 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:10.137240 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:10.137244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:10.137264 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:10.137289 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:10.139144 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:10.139575 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:10.139581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:10.139587 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:10.139729 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:10.140014 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:10.140018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:10.140020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:10.140443 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:10.140656 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:10.140942 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 17:05:10.141255 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":165,"threads_started":1}
I20260806 17:05:10.144425 23942 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader d28bb840247d47e3a039d2885030f6ec. Latest consensus state: current_term: 1 leader_uuid: "d28bb840247d47e3a039d2885030f6ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28bb840247d47e3a039d2885030f6ec" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37601 } } }
I20260806 17:05:10.144595 23942 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [sys.catalog]: This master's current role is: LEADER
I20260806 17:05:10.146107 23941 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d28bb840247d47e3a039d2885030f6ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28bb840247d47e3a039d2885030f6ec" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37601 } } }
I20260806 17:05:10.146221 23941 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [sys.catalog]: This master's current role is: LEADER
I20260806 17:05:10.146565 23945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:05:10.149984 23945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:05:10.156028 23945 catalog_manager.cc:1347] Generated new cluster ID: ef0b6ba20412452e9e38e349a8649a15
I20260806 17:05:10.156114 23945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:05:10.192173 23945 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:05:10.193207 23945 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:05:10.201148 23945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec: Generated new TSK 0
I20260806 17:05:10.201917 23945 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
I20260806 17:05:10.236931 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:37601
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:05:10.356508 23961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:10.356787 23961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:10.356832 23961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:10.361291 23961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:10.361387 23961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:10.361469 23961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:05:10.367201 23961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:37601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:10.368291 23961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:10.369297 23961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:10.379415 23970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:10.379408 23967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:10.379408 23968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:10.379678 23961 server_base.cc:1034] running on GCE node
I20260806 17:05:10.380184 23961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:10.381059 23961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:10.382288 23961 hybrid_clock.cc:648] HybridClock initialized: now 1786035910382256 us; error 55 us; skew 500 ppm
Aug 06 17:05:10 dist-test-slave-69rg krb5kdc[23848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035910, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:10.385411 23961 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:05:10.386452 23961 webserver.cc:460] Webserver started at http://127.9.111.65:33659/ using document root <none> and password file <none>
I20260806 17:05:10.387091 23961 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:10.387161 23961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:10.387342 23961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:05:10.389456 23961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "a43e12e8e3464576b6427599e1be6926"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "42b3cc230022a6f433f7c4f3379a0e77"
server_key_iv: "0a27fffc785e16259c775eaaf2f320ef"
server_key_version: "encryptionkey@0"
I20260806 17:05:10.390054 23961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "a43e12e8e3464576b6427599e1be6926"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "42b3cc230022a6f433f7c4f3379a0e77"
server_key_iv: "0a27fffc785e16259c775eaaf2f320ef"
server_key_version: "encryptionkey@0"
I20260806 17:05:10.393927 23961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:05:10.396450 23977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:10.397878 23961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 17:05:10.398041 23961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "a43e12e8e3464576b6427599e1be6926"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "42b3cc230022a6f433f7c4f3379a0e77"
server_key_iv: "0a27fffc785e16259c775eaaf2f320ef"
server_key_version: "encryptionkey@0"
I20260806 17:05:10.398216 23961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:10.428208 23961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:10.431900 23961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:10.432138 23961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:10.432821 23961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:05:10.433707 23961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:05:10.433780 23961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:10.433892 23961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:05:10.433933 23961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:10.443634 23961 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:33917
I20260806 17:05:10.443656 24095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:33917 every 8 connection(s)
I20260806 17:05:10.444832 23961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:05:10.445053 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 23961
I20260806 17:05:10.445209 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:05:10.445550 9661 external_mini_cluster.cc:1428] Setting key 6899e6092a088cde19ddeed91db0245d
Aug 06 17:05:10 dist-test-slave-69rg krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035910, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.111.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.66@KRBTEST.COM" created.
I20260806 17:05:10.460951 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:10.446860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:50049 (local address 127.9.111.126:37601)
0806 17:05:10.447014 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:10.447019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:10.448829 (+ 1810us) server_negotiation.cc:407] Connection header received
0806 17:05:10.449957 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:10.449961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:10.450011 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:10.450093 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:10.451826 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.452488 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.454128 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.454382 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.454499 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:10.457074 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:10.457077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:10.457080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:10.457115 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:10.458692 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:10.459292 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:10.459297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:10.459299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:10.459399 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:10.459784 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:10.459787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:10.459789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:10.459989 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:10.460096 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:10.460622 (+ 526us) server_negotiation.cc:299] Negotiation successful
0806 17:05:10.460772 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 17:05:10.462074 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:10.447453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:50049)
0806 17:05:10.448666 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:10.448714 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:10.449670 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:10.450311 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:10.450321 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:10.450878 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:10.451616 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:10.451639 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.452695 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.452701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:10.453986 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:10.453998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.454556 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.454560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:10.454737 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:10.455380 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:10.455398 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:10.456896 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:10.458851 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:10.458857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:10.458874 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:10.459167 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:10.459496 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:10.459499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:10.459501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:10.459682 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:10.460095 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:10.460104 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:10.460382 (+ 278us) client_negotiation.cc:769] Sending connection context
0806 17:05:10.460581 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 17:05:10.460935 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"thread_start_us":822,"threads_started":1}
I20260806 17:05:10.463469 24096 heartbeater.cc:347] Connected to a master server at 127.9.111.126:37601
I20260806 17:05:10.463711 24096 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:10.464262 24096 heartbeater.cc:511] Master 127.9.111.126:37601 requested a full tablet report, sending...
I20260806 17:05:10.466117 23896 ts_manager.cc:194] Registered new tserver with Master: a43e12e8e3464576b6427599e1be6926 (127.9.111.65:33917)
I20260806 17:05:10.467877 23896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:50049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
I20260806 17:05:10.478718 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:37601
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:05:10.595399 24101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:10.595746 24101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:10.595824 24101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:10.599989 24101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:10.600052 24101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:10.600119 24101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:05:10.605626 24101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:37601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:10.606842 24101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:10.607849 24101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:10.618031 24109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:10.618077 24107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:10.618021 24106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:10.618485 24101 server_base.cc:1034] running on GCE node
I20260806 17:05:10.618968 24101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:10.619719 24101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:10.620929 24101 hybrid_clock.cc:648] HybridClock initialized: now 1786035910620912 us; error 31 us; skew 500 ppm
Aug 06 17:05:10 dist-test-slave-69rg krb5kdc[23848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035910, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:10.624235 24101 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:05:10.625268 24101 webserver.cc:460] Webserver started at http://127.9.111.66:43295/ using document root <none> and password file <none>
I20260806 17:05:10.625931 24101 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:10.626027 24101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:10.626257 24101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:05:10.628365 24101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "9ac10b9995a6403a8d29aa601ab87e63"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "485b866cd6959ab51e78be66b17724dc"
server_key_iv: "d82ea052fac05e4e338d87ef443c2d84"
server_key_version: "encryptionkey@0"
I20260806 17:05:10.628976 24101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "9ac10b9995a6403a8d29aa601ab87e63"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "485b866cd6959ab51e78be66b17724dc"
server_key_iv: "d82ea052fac05e4e338d87ef443c2d84"
server_key_version: "encryptionkey@0"
I20260806 17:05:10.632892 24101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 17:05:10.635700 24116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:10.637207 24101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 17:05:10.637418 24101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "9ac10b9995a6403a8d29aa601ab87e63"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "485b866cd6959ab51e78be66b17724dc"
server_key_iv: "d82ea052fac05e4e338d87ef443c2d84"
server_key_version: "encryptionkey@0"
I20260806 17:05:10.637552 24101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:10.651095 24101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:10.654839 24101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:10.655066 24101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:10.655717 24101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:05:10.656603 24101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:05:10.656675 24101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:10.656750 24101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:05:10.656780 24101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:10.666126 24101 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:36489
I20260806 17:05:10.666204 24232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:36489 every 8 connection(s)
I20260806 17:05:10.667208 24101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
Aug 06 17:05:10 dist-test-slave-69rg krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035910, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:05:10.677418 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 24101
I20260806 17:05:10.677605 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:05:10.677981 9661 external_mini_cluster.cc:1428] Setting key 6271ac46fcbfb09f3452944c9b5d0ef6
I20260806 17:05:10.683357 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:10.669226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:35933 (local address 127.9.111.126:37601)
0806 17:05:10.669422 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:10.669427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:10.670348 (+ 921us) server_negotiation.cc:407] Connection header received
0806 17:05:10.671692 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:10.671695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:10.671757 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:10.671868 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:10.673102 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.673653 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.674913 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.675157 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.675274 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:10.678499 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:10.678504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:10.678507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:10.678547 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:10.680538 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:10.681228 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:10.681235 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:10.681239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:10.681357 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:10.681744 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:10.681750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:10.681754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:10.682073 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:10.682179 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:10.683037 (+ 858us) server_negotiation.cc:299] Negotiation successful
0806 17:05:10.683215 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 17:05:10.684556 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:10.669611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:35933)
0806 17:05:10.670177 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:10.670224 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:10.671470 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:10.672049 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:10.672056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:10.672443 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:10.672943 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:10.672956 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.673794 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.673797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:10.674768 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:10.674779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.675339 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.675343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:10.675514 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:10.676147 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:10.676167 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:10.678061 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:10.680719 (+ 2658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:10.680726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:10.680753 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:10.681094 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:10.681460 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:10.681462 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:10.681464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:10.681636 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:10.682226 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:10.682238 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:10.682673 (+ 435us) client_negotiation.cc:769] Sending connection context
0806 17:05:10.682936 (+ 263us) client_negotiation.cc:240] Negotiation successful
0806 17:05:10.683458 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":182,"threads_started":1}
I20260806 17:05:10.686033 24233 heartbeater.cc:347] Connected to a master server at 127.9.111.126:37601
I20260806 17:05:10.686333 24233 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:10.686916 24233 heartbeater.cc:511] Master 127.9.111.126:37601 requested a full tablet report, sending...
I20260806 17:05:10.688150 23896 ts_manager.cc:194] Registered new tserver with Master: 9ac10b9995a6403a8d29aa601ab87e63 (127.9.111.66:36489)
I20260806 17:05:10.688786 23896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:35933
WARNING: no policy specified for kudu/127.9.111.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
I20260806 17:05:10.713161 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:37601
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:05:10.830796 24239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:10.831120 24239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:10.831195 24239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:10.835377 24239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:10.835476 24239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:10.835567 24239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:05:10.841220 24239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:37601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:10.842468 24239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:10.843770 24239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:10.854152 24245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:10.854215 24246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:10.854152 24248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:10.854506 24239 server_base.cc:1034] running on GCE node
I20260806 17:05:10.854964 24239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:10.855640 24239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:10.856830 24239 hybrid_clock.cc:648] HybridClock initialized: now 1786035910856818 us; error 25 us; skew 500 ppm
Aug 06 17:05:10 dist-test-slave-69rg krb5kdc[23848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035910, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:10.860080 24239 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:05:10.861137 24239 webserver.cc:460] Webserver started at http://127.9.111.67:45083/ using document root <none> and password file <none>
I20260806 17:05:10.861717 24239 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:10.861811 24239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:10.862074 24239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:05:10.864182 24239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "3163e7d6f71e4214a8b8398f89a05a2e"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "042ed8f827281c2561abf61c87dbebf7"
server_key_iv: "d4ff289cdd3a23837a481ec7f398f6ac"
server_key_version: "encryptionkey@0"
I20260806 17:05:10.864763 24239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "3163e7d6f71e4214a8b8398f89a05a2e"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "042ed8f827281c2561abf61c87dbebf7"
server_key_iv: "d4ff289cdd3a23837a481ec7f398f6ac"
server_key_version: "encryptionkey@0"
I20260806 17:05:10.868691 24239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 17:05:10.871366 24255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:10.872676 24239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 17:05:10.872820 24239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "3163e7d6f71e4214a8b8398f89a05a2e"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "042ed8f827281c2561abf61c87dbebf7"
server_key_iv: "d4ff289cdd3a23837a481ec7f398f6ac"
server_key_version: "encryptionkey@0"
I20260806 17:05:10.872942 24239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:10.891532 24239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:10.895056 24239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:10.895290 24239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:10.896004 24239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:05:10.896893 24239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:05:10.896961 24239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:10.897030 24239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:05:10.897078 24239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:10.906991 24239 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:33181
I20260806 17:05:10.907011 24374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:33181 every 8 connection(s)
I20260806 17:05:10.907913 24239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:05:10.911320 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 24239
I20260806 17:05:10.911458 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:05:10.911741 9661 external_mini_cluster.cc:1428] Setting key 2e04f2d20d02360f4b81dc36adf1c1dd
Aug 06 17:05:10 dist-test-slave-69rg krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035910, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:05:10.921954 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:10.909733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:40271 (local address 127.9.111.126:37601)
0806 17:05:10.909927 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:10.909931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:10.910705 (+ 774us) server_negotiation.cc:407] Connection header received
0806 17:05:10.912185 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:10.912188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:10.912253 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:10.912358 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:10.913552 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.914160 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.915308 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.915532 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.915646 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:10.918269 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:10.918272 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:10.918299 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:10.918325 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:10.919765 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:10.920325 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:10.920329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:10.920330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:10.920421 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:10.920771 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:10.920776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:10.920777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:10.920976 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:10.921086 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:10.921604 (+ 518us) server_negotiation.cc:299] Negotiation successful
0806 17:05:10.921750 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 17:05:10.922879 24377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:10.910126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.67:40271)
0806 17:05:10.910558 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:10.910599 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:10.911968 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:10.912535 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:10.912542 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:10.912918 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:10.913407 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:10.913421 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.914288 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.914291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:10.915172 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:10.915182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.915690 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.915694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:10.915869 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:10.916452 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:10.916468 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:10.918084 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:10.919895 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:10.919901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:10.919939 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:10.920213 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:10.920515 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:10.920519 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:10.920521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:10.920675 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:10.921084 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:10.921091 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:10.921361 (+ 270us) client_negotiation.cc:769] Sending connection context
0806 17:05:10.921561 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 17:05:10.921958 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":122,"threads_started":1}
I20260806 17:05:10.924736 24375 heartbeater.cc:347] Connected to a master server at 127.9.111.126:37601
I20260806 17:05:10.925036 24375 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:10.925752 24375 heartbeater.cc:511] Master 127.9.111.126:37601 requested a full tablet report, sending...
I20260806 17:05:10.927028 23896 ts_manager.cc:194] Registered new tserver with Master: 3163e7d6f71e4214a8b8398f89a05a2e (127.9.111.67:33181)
I20260806 17:05:10.927688 23896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:40271
I20260806 17:05:10.938416 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 17:05:10.949154 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:10.940674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.9.111.126:37601)
0806 17:05:10.940845 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:10.940852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:10.940990 (+ 138us) server_negotiation.cc:407] Connection header received
0806 17:05:10.941274 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:10.941277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:10.941333 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:10.941426 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:10.942300 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.942836 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.943842 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.944053 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.944135 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:10.945713 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:10.945715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:10.945717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:10.945741 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:10.947377 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:10.947824 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:10.947827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:10.947828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:10.947907 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:10.948242 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:10.948244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:10.948245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:10.948486 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:10.948590 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:10.948852 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 17:05:10.948957 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 17:05:10.952342 23896 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:05:10.955539 23896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:05:10.970037 24386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:10.963915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:33917 (local address 127.0.0.1:33622)
0806 17:05:10.964657 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:10.964711 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:10.964835 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:10.965855 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:10.965861 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:10.965884 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:10.966115 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:10.966123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.967178 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.967182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:10.968532 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:10.968543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.969472 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.969476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:10.969581 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:10.969617 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 17:05:10.969731 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 17:05:10.969881 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":51,"spinlock_wait_cycles":2816,"thread_start_us":106,"threads_started":1}
I20260806 17:05:10.970041 24389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:10.964184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33622 (local address 127.9.111.65:33917)
0806 17:05:10.965134 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:10.965144 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:10.965171 (+ 27us) server_negotiation.cc:407] Connection header received
0806 17:05:10.965255 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:10.965262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:10.965489 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:10.965635 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:10.966255 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.967055 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.968659 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.969351 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.969472 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:10.969625 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:10.969774 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 17:05:10.969895 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":108,"threads_started":1}
I20260806 17:05:10.970649 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:10.964699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:36489 (local address 127.0.0.1:32772)
0806 17:05:10.965233 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:10.965246 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:10.965339 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:10.965912 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:10.965915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:10.965933 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:10.966114 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:10.966121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.967563 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.967566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:10.968729 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:10.968736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.970014 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.970017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:10.970119 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:10.970363 (+ 244us) client_negotiation.cc:769] Sending connection context
0806 17:05:10.970434 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 17:05:10.970508 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":86,"thread_start_us":109,"threads_started":1}
I20260806 17:05:10.970691 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:10.965151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32772 (local address 127.9.111.66:36489)
0806 17:05:10.965442 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:10.965455 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:10.965477 (+ 22us) server_negotiation.cc:407] Connection header received
0806 17:05:10.965552 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:10.965557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:10.965727 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:10.965935 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:10.966223 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.967408 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.968846 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.969662 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.969769 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:10.969952 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:10.970467 (+ 515us) server_negotiation.cc:299] Negotiation successful
0806 17:05:10.970562 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":116,"threads_started":1}
I20260806 17:05:10.971067 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:10.963866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:33181 (local address 127.0.0.1:37806)
0806 17:05:10.964656 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:10.964710 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:10.964837 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:10.966224 (+ 1387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:10.966227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:10.966245 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:10.966349 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:10.966355 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.967762 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.967766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:10.968863 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:10.968874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.970121 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.970125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:10.970461 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:10.970481 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 17:05:10.970544 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 17:05:10.970601 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"spinlock_wait_cycles":768,"thread_start_us":84,"threads_started":1}
I20260806 17:05:10.971596 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:10.963999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37806 (local address 127.9.111.67:33181)
0806 17:05:10.965602 (+ 1603us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:10.965615 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:10.965637 (+ 22us) server_negotiation.cc:407] Connection header received
0806 17:05:10.965707 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:10.965712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:10.965908 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:10.966055 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:10.966474 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.967579 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.968985 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:10.969766 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:10.969944 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:10.970075 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:10.970716 (+ 641us) server_negotiation.cc:299] Negotiation successful
0806 17:05:10.971461 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1480,"thread_start_us":170,"threads_started":1}
I20260806 17:05:10.973238 24028 tablet_service.cc:1467] Processing CreateTablet for tablet a39becbfffdd49c4b965b24518df20e1 (DEFAULT_TABLE table=test-table [id=0fdecabcaf664655bd2a3d5504ae9e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:05:10.973984 24306 tablet_service.cc:1467] Processing CreateTablet for tablet a39becbfffdd49c4b965b24518df20e1 (DEFAULT_TABLE table=test-table [id=0fdecabcaf664655bd2a3d5504ae9e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:05:10.973918 24165 tablet_service.cc:1467] Processing CreateTablet for tablet a39becbfffdd49c4b965b24518df20e1 (DEFAULT_TABLE table=test-table [id=0fdecabcaf664655bd2a3d5504ae9e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:05:10.974341 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39becbfffdd49c4b965b24518df20e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:10.974891 24306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39becbfffdd49c4b965b24518df20e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:10.974885 24165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39becbfffdd49c4b965b24518df20e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:10.980616 24392 tablet_bootstrap.cc:492] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926: Bootstrap starting.
I20260806 17:05:10.982462 24394 tablet_bootstrap.cc:492] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e: Bootstrap starting.
I20260806 17:05:10.983752 24392 tablet_bootstrap.cc:654] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:10.984493 24394 tablet_bootstrap.cc:654] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:10.984495 24392 log.cc:824] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:10.985146 24393 tablet_bootstrap.cc:492] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63: Bootstrap starting.
I20260806 17:05:10.985265 24394 log.cc:824] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:10.986738 24392 tablet_bootstrap.cc:492] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926: No bootstrap required, opened a new log
I20260806 17:05:10.986953 24392 ts_tablet_manager.cc:1397] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 17:05:10.987160 24394 tablet_bootstrap.cc:492] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e: No bootstrap required, opened a new log
I20260806 17:05:10.987332 24394 ts_tablet_manager.cc:1397] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 17:05:10.987591 24393 tablet_bootstrap.cc:654] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:10.988304 24393 log.cc:824] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:10.989933 24393 tablet_bootstrap.cc:492] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63: No bootstrap required, opened a new log
I20260806 17:05:10.990119 24393 ts_tablet_manager.cc:1397] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 17:05:10.991303 24394 raft_consensus.cc:348] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } }
I20260806 17:05:10.991621 24394 raft_consensus.cc:374] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:10.991470 24392 raft_consensus.cc:348] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } }
I20260806 17:05:10.991688 24392 raft_consensus.cc:374] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:10.991681 24394 raft_consensus.cc:729] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3163e7d6f71e4214a8b8398f89a05a2e, State: Initialized, Role: FOLLOWER
I20260806 17:05:10.991736 24392 raft_consensus.cc:729] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43e12e8e3464576b6427599e1be6926, State: Initialized, Role: FOLLOWER
I20260806 17:05:10.992189 24392 consensus_queue.cc:260] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } }
I20260806 17:05:10.992189 24394 consensus_queue.cc:260] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } }
I20260806 17:05:10.993189 24375 heartbeater.cc:503] Master 127.9.111.126:37601 was elected leader, sending a full tablet report...
I20260806 17:05:10.993286 24394 ts_tablet_manager.cc:1428] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260806 17:05:10.993280 24392 ts_tablet_manager.cc:1428] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 17:05:10.993489 24393 raft_consensus.cc:348] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } }
I20260806 17:05:10.993760 24393 raft_consensus.cc:374] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:10.993847 24393 raft_consensus.cc:729] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac10b9995a6403a8d29aa601ab87e63, State: Initialized, Role: FOLLOWER
I20260806 17:05:10.994199 24096 heartbeater.cc:503] Master 127.9.111.126:37601 was elected leader, sending a full tablet report...
I20260806 17:05:10.994488 24393 consensus_queue.cc:260] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } }
I20260806 17:05:10.995586 24393 ts_tablet_manager.cc:1428] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260806 17:05:10.995944 24233 heartbeater.cc:503] Master 127.9.111.126:37601 was elected leader, sending a full tablet report...
I20260806 17:05:11.042660 24400 raft_consensus.cc:482] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:05:11.042910 24400 raft_consensus.cc:504] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } }
I20260806 17:05:11.044242 24400 leader_election.cc:288] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3163e7d6f71e4214a8b8398f89a05a2e (127.9.111.67:33181), a43e12e8e3464576b6427599e1be6926 (127.9.111.65:33917)
I20260806 17:05:11.048334 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:11.044704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:34843 (local address 127.9.111.67:33181)
0806 17:05:11.044854 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:11.044858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:11.044876 (+ 18us) server_negotiation.cc:407] Connection header received
0806 17:05:11.044935 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:11.044939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:11.044994 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:11.045091 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:11.045412 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.046225 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.047295 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.047728 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.047877 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:11.047942 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:11.048120 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 17:05:11.048196 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 17:05:11.048334 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:11.044491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:33181 (local address 127.9.111.66:34843)
0806 17:05:11.044683 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:11.044700 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:11.044802 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:11.045104 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:11.045108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:11.045130 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:11.045277 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:11.045284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.046350 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.046352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:11.047158 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:11.047167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.047871 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.047874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:11.047993 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:11.048012 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 17:05:11.048101 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 17:05:11.048184 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":1}
I20260806 17:05:11.048556 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:11.044525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:33917 (local address 127.9.111.66:54571)
0806 17:05:11.045017 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:11.045034 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:11.045155 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:11.045572 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:11.045574 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:11.045592 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:11.045695 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:11.045700 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.046619 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.046625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:11.047395 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:11.047404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.048105 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.048108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:11.048223 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:11.048241 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 17:05:11.048337 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 17:05:11.048409 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":148,"threads_started":1}
I20260806 17:05:11.048543 24389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:11.045041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:54571 (local address 127.9.111.65:33917)
0806 17:05:11.045301 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:11.045305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:11.045317 (+ 12us) server_negotiation.cc:407] Connection header received
0806 17:05:11.045389 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:11.045393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:11.045438 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:11.045523 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:11.045850 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.046505 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.047530 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.047976 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.048108 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:11.048164 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:11.048354 (+ 190us) server_negotiation.cc:299] Negotiation successful
0806 17:05:11.048426 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260806 17:05:11.049280 24050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39becbfffdd49c4b965b24518df20e1" candidate_uuid: "9ac10b9995a6403a8d29aa601ab87e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a43e12e8e3464576b6427599e1be6926" is_pre_election: true
I20260806 17:05:11.049280 24327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39becbfffdd49c4b965b24518df20e1" candidate_uuid: "9ac10b9995a6403a8d29aa601ab87e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" is_pre_election: true
I20260806 17:05:11.049640 24327 raft_consensus.cc:2454] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ac10b9995a6403a8d29aa601ab87e63 in term 0.
I20260806 17:05:11.049640 24050 raft_consensus.cc:2454] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ac10b9995a6403a8d29aa601ab87e63 in term 0.
I20260806 17:05:11.050269 24118 leader_election.cc:302] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3163e7d6f71e4214a8b8398f89a05a2e, 9ac10b9995a6403a8d29aa601ab87e63, a43e12e8e3464576b6427599e1be6926; no voters:
I20260806 17:05:11.050598 24400 raft_consensus.cc:2781] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:05:11.050684 24400 raft_consensus.cc:482] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:05:11.050714 24400 raft_consensus.cc:3037] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:11.051817 24400 raft_consensus.cc:504] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } }
I20260806 17:05:11.052188 24400 leader_election.cc:288] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [CANDIDATE]: Term 1 election: Requested vote from peers 3163e7d6f71e4214a8b8398f89a05a2e (127.9.111.67:33181), a43e12e8e3464576b6427599e1be6926 (127.9.111.65:33917)
I20260806 17:05:11.052642 24327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39becbfffdd49c4b965b24518df20e1" candidate_uuid: "9ac10b9995a6403a8d29aa601ab87e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3163e7d6f71e4214a8b8398f89a05a2e"
I20260806 17:05:11.052640 24050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39becbfffdd49c4b965b24518df20e1" candidate_uuid: "9ac10b9995a6403a8d29aa601ab87e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a43e12e8e3464576b6427599e1be6926"
I20260806 17:05:11.052793 24050 raft_consensus.cc:3037] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:11.052798 24327 raft_consensus.cc:3037] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:11.053905 24327 raft_consensus.cc:2454] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ac10b9995a6403a8d29aa601ab87e63 in term 1.
I20260806 17:05:11.053910 24050 raft_consensus.cc:2454] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ac10b9995a6403a8d29aa601ab87e63 in term 1.
I20260806 17:05:11.054211 24118 leader_election.cc:302] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac10b9995a6403a8d29aa601ab87e63, a43e12e8e3464576b6427599e1be6926; no voters:
I20260806 17:05:11.054400 24400 raft_consensus.cc:2781] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:05:11.054608 24400 raft_consensus.cc:686] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [term 1 LEADER]: Becoming Leader. State: Replica: 9ac10b9995a6403a8d29aa601ab87e63, State: Running, Role: LEADER
I20260806 17:05:11.054982 24400 consensus_queue.cc:237] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } }
I20260806 17:05:11.058139 23896 catalog_manager.cc:5495] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ac10b9995a6403a8d29aa601ab87e63 (127.9.111.66). New cstate: current_term: 1 leader_uuid: "9ac10b9995a6403a8d29aa601ab87e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } health_report { overall_health: HEALTHY } } }
I20260806 17:05:11.072039 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:11.068299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.9.111.66:36489)
0806 17:05:11.068450 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:11.068455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:11.068470 (+ 15us) server_negotiation.cc:407] Connection header received
0806 17:05:11.068648 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:11.068651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:11.068708 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:11.068841 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:05:11.069082 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.069583 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.070524 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.070745 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.070857 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:11.071179 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:05:11.071605 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:05:11.071733 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:11.071807 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 17:05:11.071867 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 17:05:11.078334 24050 raft_consensus.cc:1261] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Refusing update from remote peer 9ac10b9995a6403a8d29aa601ab87e63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:05:11.078334 24327 raft_consensus.cc:1261] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e [term 1 FOLLOWER]: Refusing update from remote peer 9ac10b9995a6403a8d29aa601ab87e63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:05:11.079084 24400 consensus_queue.cc:1035] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:05:11.079304 24402 consensus_queue.cc:1035] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:05:11.093215 24405 mvcc.cc:205] Tried to move back new op lower bound from 7315603091768385536 to 7315603091683872768. Current Snapshot: MvccSnapshot[applied={T|T < 7315603091768385536}]
I20260806 17:05:11.096408 24407 mvcc.cc:205] Tried to move back new op lower bound from 7315603091768385536 to 7315603091683872768. Current Snapshot: MvccSnapshot[applied={T|T < 7315603091768385536}]
I20260806 17:05:11.102150 23896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:05:11.102370 23896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:05:11.104979 23896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec: Sending DeleteTablet for 3 replicas of tablet a39becbfffdd49c4b965b24518df20e1
I20260806 17:05:11.105813 24306 tablet_service.cc:1514] Processing DeleteTablet for tablet a39becbfffdd49c4b965b24518df20e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:05:11 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:37806
I20260806 17:05:11.105937 24165 tablet_service.cc:1514] Processing DeleteTablet for tablet a39becbfffdd49c4b965b24518df20e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:05:11 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:32772
I20260806 17:05:11.106032 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet a39becbfffdd49c4b965b24518df20e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:05:11 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:33622
I20260806 17:05:11.106549 24417 tablet_replica.cc:330] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e: stopping tablet replica
I20260806 17:05:11.106707 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 23961
I20260806 17:05:11.108147 24417 raft_consensus.cc:2229] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:05:11.108500 24417 raft_consensus.cc:2258] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:05:11.110021 24416 tablet_replica.cc:330] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63: stopping tablet replica
I20260806 17:05:11.110255 24417 ts_tablet_manager.cc:1905] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:05:11.110463 24416 raft_consensus.cc:2229] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:05:11.110884 24416 raft_consensus.cc:2258] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:05:11.112289 24416 ts_tablet_manager.cc:1905] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:05:11.114060 24417 ts_tablet_manager.cc:1918] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:05:11.114212 24417 log.cc:1196] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/a39becbfffdd49c4b965b24518df20e1
I20260806 17:05:11.114605 24417 ts_tablet_manager.cc:1939] T a39becbfffdd49c4b965b24518df20e1 P 3163e7d6f71e4214a8b8398f89a05a2e: Deleting consensus metadata
I20260806 17:05:11.115723 24416 ts_tablet_manager.cc:1918] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:05:11.115931 24416 log.cc:1196] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/a39becbfffdd49c4b965b24518df20e1
I20260806 17:05:11.116256 23879 catalog_manager.cc:4841] TS 3163e7d6f71e4214a8b8398f89a05a2e (127.9.111.67:33181): tablet a39becbfffdd49c4b965b24518df20e1 (table test-table [id=0fdecabcaf664655bd2a3d5504ae9e90]) successfully deleted
I20260806 17:05:11.116354 24416 ts_tablet_manager.cc:1939] T a39becbfffdd49c4b965b24518df20e1 P 9ac10b9995a6403a8d29aa601ab87e63: Deleting consensus metadata
W20260806 17:05:11.116959 23880 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.111.65:33917 (error 108)
I20260806 17:05:11.118086 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:11.117742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:33917 (local address 127.0.0.1:33626)
0806 17:05:11.117900 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:11.117980 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.65:33917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 17:05:11.118244 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 24101
W20260806 17:05:11.118300 23880 catalog_manager.cc:4584] TS a43e12e8e3464576b6427599e1be6926 (127.9.111.65:33917): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a39becbfffdd49c4b965b24518df20e1: Network error: Client connection negotiation failed: client connection to 127.9.111.65:33917: connect: Connection refused (error 111)
I20260806 17:05:11.127491 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 24239
I20260806 17:05:11.129084 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:11.128709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:36489 (local address 127.0.0.1:32784)
0806 17:05:11.128931 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:11.128989 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.66:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260806 17:05:11.136117 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 23862
I20260806 17:05:11.144402 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:37601
--webserver_interface=127.9.111.126
--webserver_port=33425
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:37601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:05:11.260646 24420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:11.261122 24420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:11.261322 24420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:11.265568 24420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:05:11.265672 24420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:11.265713 24420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:11.265750 24420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:05:11.265790 24420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:05:11.271415 24420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:37601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:37601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=33425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:11.272784 24420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:11.274186 24420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:11.287930 24427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:11.287957 24426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:11.287974 24429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:11.288291 24420 server_base.cc:1034] running on GCE node
I20260806 17:05:11.288919 24420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:11.290195 24420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:11.291404 24420 hybrid_clock.cc:648] HybridClock initialized: now 1786035911291365 us; error 46 us; skew 500 ppm
Aug 06 17:05:11 dist-test-slave-69rg krb5kdc[23848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035911, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:11.294477 24420 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:05:11.295502 24420 webserver.cc:460] Webserver started at http://127.9.111.126:33425/ using document root <none> and password file <none>
I20260806 17:05:11.296113 24420 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:11.296223 24420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:11.300331 24420 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 17:05:11.302716 24436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:11.304083 24420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 17:05:11.304241 24420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "d28bb840247d47e3a039d2885030f6ec"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "555b15fbde0b497b35a76cb59ed6b938"
server_key_iv: "5a8e6ca5ef3df8c8e076fc8039f15ba3"
server_key_version: "encryptionkey@0"
I20260806 17:05:11.304878 24420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:11.329206 24420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:11.332993 24420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:11.333225 24420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:11.340998 24420 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:37601
I20260806 17:05:11.341295 24490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:37601 every 8 connection(s)
I20260806 17:05:11.342242 24420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:05:11.342674 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 24420
I20260806 17:05:11.343192 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:33917
--local_ip_for_outbound_sockets=127.9.111.65
--tserver_master_addrs=127.9.111.126:37601
--webserver_port=33659
--webserver_interface=127.9.111.65
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:05:11.347803 24491 sys_catalog.cc:263] Verifying existing consensus state
I20260806 17:05:11.350719 24491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec: Bootstrap starting.
I20260806 17:05:11.374979 24491 log.cc:824] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:11.386525 24491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 17:05:11.386982 24491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec: Bootstrap complete.
I20260806 17:05:11.391418 24491 raft_consensus.cc:348] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28bb840247d47e3a039d2885030f6ec" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37601 } }
I20260806 17:05:11.391863 24491 raft_consensus.cc:729] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d28bb840247d47e3a039d2885030f6ec, State: Initialized, Role: FOLLOWER
I20260806 17:05:11.392413 24491 consensus_queue.cc:260] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28bb840247d47e3a039d2885030f6ec" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37601 } }
I20260806 17:05:11.392561 24491 raft_consensus.cc:388] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:05:11.392658 24491 raft_consensus.cc:482] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:05:11.392776 24491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 1 FOLLOWER]: Advancing to term 2
I20260806 17:05:11.394948 24491 raft_consensus.cc:504] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28bb840247d47e3a039d2885030f6ec" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37601 } }
I20260806 17:05:11.395301 24491 leader_election.cc:302] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d28bb840247d47e3a039d2885030f6ec; no voters:
I20260806 17:05:11.395637 24491 leader_election.cc:288] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 17:05:11.395780 24495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 2 FOLLOWER]: Leader election won for term 2
I20260806 17:05:11.396029 24495 raft_consensus.cc:686] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 2 LEADER]: Becoming Leader. State: Replica: d28bb840247d47e3a039d2885030f6ec, State: Running, Role: LEADER
I20260806 17:05:11.396354 24495 consensus_queue.cc:237] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28bb840247d47e3a039d2885030f6ec" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37601 } }
I20260806 17:05:11.396898 24491 sys_catalog.cc:564] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:05:11.397387 24495 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader d28bb840247d47e3a039d2885030f6ec. Latest consensus state: current_term: 2 leader_uuid: "d28bb840247d47e3a039d2885030f6ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28bb840247d47e3a039d2885030f6ec" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37601 } } }
I20260806 17:05:11.397505 24495 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [sys.catalog]: This master's current role is: LEADER
I20260806 17:05:11.398177 24496 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d28bb840247d47e3a039d2885030f6ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28bb840247d47e3a039d2885030f6ec" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37601 } } }
I20260806 17:05:11.398288 24496 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [sys.catalog]: This master's current role is: LEADER
I20260806 17:05:11.399086 24499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:05:11.403998 24499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:05:11.404857 24499 catalog_manager.cc:1259] Loaded cluster ID: ef0b6ba20412452e9e38e349a8649a15
I20260806 17:05:11.404930 24499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:05:11.406467 24499 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 17:05:11.410972 24499 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 17:05:11.416801 24499 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:05:11.417768 24499 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec: Loaded TSK: 0
I20260806 17:05:11.419098 24499 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 17:05:11.479926 24493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:11.480224 24493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:11.480271 24493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:11.484768 24493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:11.484843 24493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:11.484906 24493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:05:11.490581 24493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:33917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=33659
--tserver_master_addrs=127.9.111.126:37601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:11.491753 24493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:11.492825 24493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:11.502915 24521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:11.502915 24520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:11.503445 24523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:11.503473 24493 server_base.cc:1034] running on GCE node
I20260806 17:05:11.503899 24493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:11.504622 24493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:11.505868 24493 hybrid_clock.cc:648] HybridClock initialized: now 1786035911505795 us; error 49 us; skew 500 ppm
Aug 06 17:05:11 dist-test-slave-69rg krb5kdc[23848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035911, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:11.509151 24493 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:05:11.510324 24493 webserver.cc:460] Webserver started at http://127.9.111.65:33659/ using document root <none> and password file <none>
I20260806 17:05:11.510929 24493 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:11.511044 24493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:11.515517 24493 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 17:05:11.518004 24530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:11.519542 24493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:05:11.519748 24493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "a43e12e8e3464576b6427599e1be6926"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "42b3cc230022a6f433f7c4f3379a0e77"
server_key_iv: "0a27fffc785e16259c775eaaf2f320ef"
server_key_version: "encryptionkey@0"
I20260806 17:05:11.520404 24493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:11.542289 24493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:11.545984 24493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:11.546212 24493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:11.546870 24493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:05:11.548041 24537 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 17:05:11.552359 24493 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 17:05:11.552485 24493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260806 17:05:11.552572 24493 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 17:05:11.554383 24493 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 17:05:11.554461 24493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260806 17:05:11.554786 24537 tablet_bootstrap.cc:492] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926: Bootstrap starting.
I20260806 17:05:11.565567 24493 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:33917
I20260806 17:05:11.565716 24647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:33917 every 8 connection(s)
I20260806 17:05:11.566627 24493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:05:11.572631 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 24493
I20260806 17:05:11.573113 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:36489
--local_ip_for_outbound_sockets=127.9.111.66
--tserver_master_addrs=127.9.111.126:37601
--webserver_port=43295
--webserver_interface=127.9.111.66
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:05:11.574841 24537 log.cc:824] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926: Log is configured to *not* fsync() on all Append() calls
Aug 06 17:05:11 dist-test-slave-69rg krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035911, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:05:11.583034 24537 tablet_bootstrap.cc:492] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 17:05:11.583509 24537 tablet_bootstrap.cc:492] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926: Bootstrap complete.
I20260806 17:05:11.584872 24537 ts_tablet_manager.cc:1397] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.004s
I20260806 17:05:11.587213 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:11.569116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:47901 (local address 127.9.111.126:37601)
0806 17:05:11.569734 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:11.569754 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:11.569803 (+ 49us) server_negotiation.cc:407] Connection header received
0806 17:05:11.570972 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:11.570996 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:11.571424 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:11.571832 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:11.573371 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.574430 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.576095 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.576798 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.576975 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:11.580062 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:11.580071 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:11.580092 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:11.580139 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:11.582735 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:11.583451 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:11.583461 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:11.583470 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:11.583673 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:11.584097 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:11.584102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:11.584105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:11.584662 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:11.584830 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:11.585657 (+ 827us) server_negotiation.cc:299] Negotiation successful
0806 17:05:11.586100 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":158,"threads_started":1}
I20260806 17:05:11.587335 24537 raft_consensus.cc:348] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } }
I20260806 17:05:11.587924 24537 raft_consensus.cc:729] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43e12e8e3464576b6427599e1be6926, State: Initialized, Role: FOLLOWER
I20260806 17:05:11.588435 24537 consensus_queue.cc:260] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } }
I20260806 17:05:11.588811 24650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:11.569071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:47901)
0806 17:05:11.569560 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:11.569617 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:11.570462 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:11.571894 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:11.571901 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:11.572494 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:11.573024 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:11.573042 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.574599 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.574602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:11.575927 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:11.575943 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.576942 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.576945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:11.577084 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:11.577733 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:11.577754 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:11.579808 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:11.582896 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:11.582902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:11.582915 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:11.583293 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:11.583788 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:11.583794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:11.583799 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:11.583982 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:11.584902 (+ 920us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:11.584912 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:11.585252 (+ 340us) client_negotiation.cc:769] Sending connection context
0806 17:05:11.587518 (+ 2266us) client_negotiation.cc:240] Negotiation successful
0806 17:05:11.587912 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":155,"threads_started":1}
I20260806 17:05:11.589551 24537 ts_tablet_manager.cc:1428] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260806 17:05:11.591405 24648 heartbeater.cc:347] Connected to a master server at 127.9.111.126:37601
I20260806 17:05:11.591665 24648 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:11.592176 24648 heartbeater.cc:511] Master 127.9.111.126:37601 requested a full tablet report, sending...
I20260806 17:05:11.594792 24449 ts_manager.cc:194] Registered new tserver with Master: a43e12e8e3464576b6427599e1be6926 (127.9.111.65:33917)
I20260806 17:05:11.598483 24449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:47901
I20260806 17:05:11.601183 24648 heartbeater.cc:503] Master 127.9.111.126:37601 was elected leader, sending a full tablet report...
Aug 06 17:05:11 dist-test-slave-69rg krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035911, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for kudu/127.9.111.65@KRBTEST.COM
I20260806 17:05:11.610463 24656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:11.598047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33640 (local address 127.9.111.65:33917)
0806 17:05:11.598570 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:11.598587 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:11.599070 (+ 483us) server_negotiation.cc:407] Connection header received
0806 17:05:11.599284 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:11.599291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:11.599513 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:11.599658 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:11.600397 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.601469 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.602823 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.603259 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.603378 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:11.606052 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:11.606064 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:11.606078 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:11.606111 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:11.608085 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:11.608674 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:11.608682 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:11.608684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:11.608798 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:11.609234 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:11.609237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:11.609238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:11.609523 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:11.609711 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:11.610113 (+ 402us) server_negotiation.cc:299] Negotiation successful
0806 17:05:11.610309 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":156,"threads_started":1}
I20260806 17:05:11.610463 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:11.597912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:33917 (local address 127.0.0.1:33640)
0806 17:05:11.598912 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:11.598956 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:11.599110 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:11.599674 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:11.599679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:11.600124 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:11.600256 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:11.600265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.601747 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.601750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:11.602673 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:11.602683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.603384 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.603388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:11.603501 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:11.604169 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:11.604181 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:11.605889 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:11.608222 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:11.608235 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:11.608239 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:11.608516 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:11.608913 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:11.608916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:11.608917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:11.609035 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:11.609671 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:11.609677 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:11.609857 (+ 180us) client_negotiation.cc:769] Sending connection context
0806 17:05:11.610006 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 17:05:11.610252 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":159,"threads_started":1}
I20260806 17:05:11.611646 24581 tablet_service.cc:1514] Processing DeleteTablet for tablet a39becbfffdd49c4b965b24518df20e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:05:11 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:33640
I20260806 17:05:11.612783 24657 tablet_replica.cc:330] stopping tablet replica
I20260806 17:05:11.613148 24657 raft_consensus.cc:2229] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:05:11.613415 24657 raft_consensus.cc:2258] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:05:11.614483 24657 ts_tablet_manager.cc:1905] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:05:11.618661 24657 ts_tablet_manager.cc:1918] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:05:11.618831 24657 log.cc:1196] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/wal/wals/a39becbfffdd49c4b965b24518df20e1
I20260806 17:05:11.619127 24657 ts_tablet_manager.cc:1939] T a39becbfffdd49c4b965b24518df20e1 P a43e12e8e3464576b6427599e1be6926: Deleting consensus metadata
I20260806 17:05:11.621053 24438 catalog_manager.cc:4841] TS a43e12e8e3464576b6427599e1be6926 (127.9.111.65:33917): tablet a39becbfffdd49c4b965b24518df20e1 (table test-table [id=0fdecabcaf664655bd2a3d5504ae9e90]) successfully deleted
W20260806 17:05:11.700511 24652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:11.700773 24652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:11.700819 24652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:11.705080 24652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:11.705163 24652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:11.705224 24652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:05:11.711109 24652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:36489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=43295
--tserver_master_addrs=127.9.111.126:37601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:11.712334 24652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:11.713423 24652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:11.723783 24666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:11.723791 24664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:11.723822 24663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:11.724073 24652 server_base.cc:1034] running on GCE node
I20260806 17:05:11.724712 24652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:11.725461 24652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:11.726668 24652 hybrid_clock.cc:648] HybridClock initialized: now 1786035911726645 us; error 36 us; skew 500 ppm
Aug 06 17:05:11 dist-test-slave-69rg krb5kdc[23848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035911, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:11.729579 24652 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:05:11.730594 24652 webserver.cc:460] Webserver started at http://127.9.111.66:43295/ using document root <none> and password file <none>
I20260806 17:05:11.731139 24652 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:11.731216 24652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:11.735208 24652 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260806 17:05:11.737246 24674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:11.738627 24652 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 17:05:11.738778 24652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "9ac10b9995a6403a8d29aa601ab87e63"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "485b866cd6959ab51e78be66b17724dc"
server_key_iv: "d82ea052fac05e4e338d87ef443c2d84"
server_key_version: "encryptionkey@0"
I20260806 17:05:11.739418 24652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:11.757222 24652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:11.760645 24652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:11.760825 24652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:11.761418 24652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:05:11.762352 24652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:05:11.762427 24652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:11.762472 24652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:05:11.762488 24652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:11.771996 24652 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:36489
I20260806 17:05:11.772019 24789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:36489 every 8 connection(s)
I20260806 17:05:11.773092 24652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
Aug 06 17:05:11 dist-test-slave-69rg krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035911, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:05:11.782922 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 24652
I20260806 17:05:11.783377 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:33181
--local_ip_for_outbound_sockets=127.9.111.67
--tserver_master_addrs=127.9.111.126:37601
--webserver_port=45083
--webserver_interface=127.9.111.67
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:05:11.787752 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:11.774959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:43839 (local address 127.9.111.126:37601)
0806 17:05:11.775102 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:11.775106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:11.775887 (+ 781us) server_negotiation.cc:407] Connection header received
0806 17:05:11.777209 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:11.777212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:11.777276 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:11.777386 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:11.778664 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.779190 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.780318 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.780554 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.780666 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:11.783341 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:11.783346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:11.783349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:11.783378 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:11.785130 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:11.785880 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:11.785883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:11.785885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:11.785971 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:11.786379 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:11.786382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:11.786383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:11.786570 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:11.786648 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:11.787325 (+ 677us) server_negotiation.cc:299] Negotiation successful
0806 17:05:11.787496 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 17:05:11.788415 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:11.775290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:43839)
0806 17:05:11.775731 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:11.775773 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:11.776996 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:11.777570 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:11.777577 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:11.778033 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:11.778519 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:11.778532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.779318 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.779322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:11.780190 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:11.780199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.780706 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.780710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:11.780878 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:11.781469 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:11.781485 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:11.783159 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:11.785253 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:11.785258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:11.785272 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:11.785725 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:11.786067 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:11.786072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:11.786075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:11.786278 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:11.786677 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:11.786686 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:11.787044 (+ 358us) client_negotiation.cc:769] Sending connection context
0806 17:05:11.787209 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 17:05:11.787481 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":120,"threads_started":1}
I20260806 17:05:11.789695 24790 heartbeater.cc:347] Connected to a master server at 127.9.111.126:37601
I20260806 17:05:11.789995 24790 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:11.790499 24790 heartbeater.cc:511] Master 127.9.111.126:37601 requested a full tablet report, sending...
I20260806 17:05:11.791584 24449 ts_manager.cc:194] Registered new tserver with Master: 9ac10b9995a6403a8d29aa601ab87e63 (127.9.111.66:36489)
I20260806 17:05:11.792168 24449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:43839
W20260806 17:05:11.902384 24794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:11.902683 24794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:11.902768 24794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:11.907048 24794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:11.907130 24794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:11.907219 24794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:05:11.913249 24794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:33181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=45083
--tserver_master_addrs=127.9.111.126:37601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:11.914824 24794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:11.915993 24794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:11.927258 24802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:11.927266 24800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:11.927266 24799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:11.927714 24794 server_base.cc:1034] running on GCE node
I20260806 17:05:11.928256 24794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:11.929085 24794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:11.930290 24794 hybrid_clock.cc:648] HybridClock initialized: now 1786035911930274 us; error 29 us; skew 500 ppm
Aug 06 17:05:11 dist-test-slave-69rg krb5kdc[23848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035911, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:11.933336 24794 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:05:11.934437 24794 webserver.cc:460] Webserver started at http://127.9.111.67:45083/ using document root <none> and password file <none>
I20260806 17:05:11.935048 24794 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:11.935150 24794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:11.939615 24794 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 17:05:11.941975 24810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:11.943480 24794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:05:11.943637 24794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "3163e7d6f71e4214a8b8398f89a05a2e"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "042ed8f827281c2561abf61c87dbebf7"
server_key_iv: "d4ff289cdd3a23837a481ec7f398f6ac"
server_key_version: "encryptionkey@0"
I20260806 17:05:11.944345 24794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:11.970320 24794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:11.974301 24794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:11.974558 24794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:11.975658 24794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:05:11.976783 24794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:05:11.976845 24794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:11.976891 24794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:05:11.976961 24794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:11.987144 24794 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:33181
I20260806 17:05:11.987171 24928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:33181 every 8 connection(s)
I20260806 17:05:11.988298 24794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:05:11.992959 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 24794
Aug 06 17:05:11 dist-test-slave-69rg krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035911, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:05:12.006054 24933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:11.994097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.9.111.126:37601)
0806 17:05:11.994547 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:11.994553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:11.994571 (+ 18us) server_negotiation.cc:407] Connection header received
0806 17:05:11.994625 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:11.994631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:11.994678 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:11.994785 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:11.995601 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.996171 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.997711 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.998737 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.998894 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:12.001354 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:12.001358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:12.001360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:12.001392 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:12.003573 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:12.004115 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:12.004119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:12.004121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:12.004215 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:12.005016 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:12.005023 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:12.005025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:12.005237 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:12.005493 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:12.005661 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 17:05:12.005810 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":351,"thread_start_us":110,"threads_started":1}
I20260806 17:05:12.006610 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:11.990604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:38019 (local address 127.9.111.126:37601)
0806 17:05:11.990817 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:11.990822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:11.991975 (+ 1153us) server_negotiation.cc:407] Connection header received
0806 17:05:11.993734 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:11.993740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:11.993810 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:11.993918 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:11.995503 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.996076 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.997531 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.998389 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.998492 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:12.002471 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:12.002475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:12.002477 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:12.002650 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:12.004215 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:12.004802 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:12.004805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:12.004806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:12.004887 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:12.005291 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:12.005294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:12.005296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:12.005496 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:12.005603 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:12.006321 (+ 718us) server_negotiation.cc:299] Negotiation successful
0806 17:05:12.006458 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260806 17:05:12.008595 24931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:11.991025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.67:38019)
0806 17:05:11.991671 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:11.991722 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:11.993286 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:11.994244 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:11.994256 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:11.994745 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:11.995330 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:11.995349 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.996237 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.996242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:11.997319 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:11.997337 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:11.998674 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:11.998684 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:11.999009 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:11.999833 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:11.999852 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:12.002198 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:12.004358 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:12.004365 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:12.004381 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:12.004685 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:12.005006 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:12.005010 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:12.005012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:12.005188 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:12.005609 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:12.005620 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:12.005989 (+ 369us) client_negotiation.cc:769] Sending connection context
0806 17:05:12.006792 (+ 803us) client_negotiation.cc:240] Negotiation successful
0806 17:05:12.007162 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":179,"threads_started":1}
I20260806 17:05:12.010015 24929 heartbeater.cc:347] Connected to a master server at 127.9.111.126:37601
I20260806 17:05:12.010294 24929 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:12.010842 24929 heartbeater.cc:511] Master 127.9.111.126:37601 requested a full tablet report, sending...
I20260806 17:05:12.012213 24449 ts_manager.cc:194] Registered new tserver with Master: 3163e7d6f71e4214a8b8398f89a05a2e (127.9.111.67:33181)
I20260806 17:05:12.013168 24449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:38019
I20260806 17:05:12.022111 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 17:05:12.027797 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:12.023253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51432 (local address 127.9.111.126:37601)
0806 17:05:12.023495 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:12.023499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:12.023517 (+ 18us) server_negotiation.cc:407] Connection header received
0806 17:05:12.023610 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:12.023612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:12.023749 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:12.023911 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:05:12.024309 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.024907 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.026011 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.026251 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.026376 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:12.026775 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:05:12.027342 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:05:12.027487 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:12.027550 (+ 63us) server_negotiation.cc:299] Negotiation successful
0806 17:05:12.027617 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 17:05:12.028612 24449 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:05:12.032258 24449 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:05:12.042706 24581 tablet_service.cc:1467] Processing CreateTablet for tablet 76edeb977b3e40cfa5f89788dac0c248 (DEFAULT_TABLE table=test-table [id=9381837fc4a5461184208a7d562e962e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:05:12.043560 24581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76edeb977b3e40cfa5f89788dac0c248. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:12.046975 24537 tablet_bootstrap.cc:492] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926: Bootstrap starting.
I20260806 17:05:12.047706 24939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:12.041419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:36489 (local address 127.0.0.1:32792)
0806 17:05:12.041780 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:12.041795 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:12.041923 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:12.042725 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:12.042728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:12.042762 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:12.042991 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:12.042999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.044442 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.044445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:12.045783 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:12.045796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.047188 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.047192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:12.047298 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:12.047327 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 17:05:12.047388 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 17:05:12.047479 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":35,"spinlock_wait_cycles":1792,"thread_start_us":142,"threads_started":1}
I20260806 17:05:12.047705 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:12.041079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:33181 (local address 127.0.0.1:37810)
0806 17:05:12.041275 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:12.041292 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:12.041376 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:12.042669 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:12.042673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:12.042695 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:12.042942 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:12.042951 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.044746 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.044749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:12.045861 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:12.045873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.047076 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.047079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:12.047209 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:12.047291 (+ 82us) client_negotiation.cc:769] Sending connection context
0806 17:05:12.047365 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 17:05:12.047476 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":67,"spinlock_wait_cycles":768}
I20260806 17:05:12.048282 24537 tablet_bootstrap.cc:654] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:12.048542 24940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:12.041431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.9.111.67:33181)
0806 17:05:12.042043 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:12.042058 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:12.042083 (+ 25us) server_negotiation.cc:407] Connection header received
0806 17:05:12.042195 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:12.042201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:12.042459 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:12.043149 (+ 690us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:12.043339 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.044585 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.046007 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.046923 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.048046 (+ 1123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:12.048203 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:12.048264 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 17:05:12.048397 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":191,"threads_started":1}
I20260806 17:05:12.048553 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:12.041891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32792 (local address 127.9.111.66:36489)
0806 17:05:12.042210 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:12.042222 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:12.042250 (+ 28us) server_negotiation.cc:407] Connection header received
0806 17:05:12.042313 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:12.042317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:12.042551 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:12.042874 (+ 323us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:12.043116 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.044281 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.045994 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.046990 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.047664 (+ 674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:12.047852 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:12.047990 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 17:05:12.048369 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":133,"threads_started":1}
I20260806 17:05:12.050333 24537 tablet_bootstrap.cc:492] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926: No bootstrap required, opened a new log
I20260806 17:05:12.050447 24537 ts_tablet_manager.cc:1397] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 17:05:12.050969 24537 raft_consensus.cc:348] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } }
I20260806 17:05:12.051112 24537 raft_consensus.cc:374] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:12.051160 24537 raft_consensus.cc:729] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43e12e8e3464576b6427599e1be6926, State: Initialized, Role: FOLLOWER
I20260806 17:05:12.051303 24537 consensus_queue.cc:260] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } }
I20260806 17:05:12.051693 24537 ts_tablet_manager.cc:1428] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 17:05:12.052101 24860 tablet_service.cc:1467] Processing CreateTablet for tablet 76edeb977b3e40cfa5f89788dac0c248 (DEFAULT_TABLE table=test-table [id=9381837fc4a5461184208a7d562e962e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:05:12.052103 24723 tablet_service.cc:1467] Processing CreateTablet for tablet 76edeb977b3e40cfa5f89788dac0c248 (DEFAULT_TABLE table=test-table [id=9381837fc4a5461184208a7d562e962e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:05:12.053236 24860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76edeb977b3e40cfa5f89788dac0c248. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:12.053251 24723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76edeb977b3e40cfa5f89788dac0c248. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:12.064817 24943 tablet_bootstrap.cc:492] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63: Bootstrap starting.
I20260806 17:05:12.066300 24944 tablet_bootstrap.cc:492] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e: Bootstrap starting.
W20260806 17:05:12.068073 24649 tablet.cc:2369] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:05:12.068236 24943 tablet_bootstrap.cc:654] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:12.068812 24944 tablet_bootstrap.cc:654] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:12.069298 24943 log.cc:824] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:12.070499 24944 log.cc:824] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:12.072062 24943 tablet_bootstrap.cc:492] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63: No bootstrap required, opened a new log
I20260806 17:05:12.072304 24943 ts_tablet_manager.cc:1397] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 17:05:12.073069 24944 tablet_bootstrap.cc:492] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e: No bootstrap required, opened a new log
I20260806 17:05:12.073297 24944 ts_tablet_manager.cc:1397] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 17:05:12.076699 24943 raft_consensus.cc:348] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } }
I20260806 17:05:12.077103 24943 raft_consensus.cc:374] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:12.077173 24943 raft_consensus.cc:729] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac10b9995a6403a8d29aa601ab87e63, State: Initialized, Role: FOLLOWER
I20260806 17:05:12.077309 24944 raft_consensus.cc:348] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } }
I20260806 17:05:12.077591 24944 raft_consensus.cc:374] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:12.077651 24944 raft_consensus.cc:729] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3163e7d6f71e4214a8b8398f89a05a2e, State: Initialized, Role: FOLLOWER
I20260806 17:05:12.077708 24943 consensus_queue.cc:260] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } }
I20260806 17:05:12.078177 24944 consensus_queue.cc:260] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } }
I20260806 17:05:12.078753 24790 heartbeater.cc:503] Master 127.9.111.126:37601 was elected leader, sending a full tablet report...
I20260806 17:05:12.079042 24943 ts_tablet_manager.cc:1428] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260806 17:05:12.079077 24944 ts_tablet_manager.cc:1428] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 17:05:12.079437 24929 heartbeater.cc:503] Master 127.9.111.126:37601 was elected leader, sending a full tablet report...
I20260806 17:05:12.087479 24948 raft_consensus.cc:482] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:05:12.087743 24948 raft_consensus.cc:504] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } }
I20260806 17:05:12.089043 24948 leader_election.cc:288] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ac10b9995a6403a8d29aa601ab87e63 (127.9.111.66:36489), a43e12e8e3464576b6427599e1be6926 (127.9.111.65:33917)
I20260806 17:05:12.093695 24931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:12.089250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:36489 (local address 127.9.111.67:47159)
0806 17:05:12.089386 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:12.089405 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:12.089507 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:12.089809 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:12.089848 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:12.089872 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:12.089990 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:12.089996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.091059 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.091063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:12.092190 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:12.092204 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.093177 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.093181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:12.093312 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:12.093348 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 17:05:12.093433 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 17:05:12.093529 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 17:05:12.094195 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:12.089382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:47159 (local address 127.9.111.66:36489)
0806 17:05:12.089532 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:12.089538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:12.089557 (+ 19us) server_negotiation.cc:407] Connection header received
0806 17:05:12.089617 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:12.089620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:12.089670 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:12.089763 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:12.090158 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.090946 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.092373 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.093014 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.093164 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:12.093223 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:12.093928 (+ 705us) server_negotiation.cc:299] Negotiation successful
0806 17:05:12.094030 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 17:05:12.095069 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:12.089567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:33917 (local address 127.9.111.67:60565)
0806 17:05:12.090065 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:12.090101 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:12.090223 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:12.090518 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:12.090521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:12.090541 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:12.090668 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:12.090674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.091582 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.091584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:12.092889 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:12.092903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.094603 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.094607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:12.094719 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:12.094747 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 17:05:12.094816 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 17:05:12.094899 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":176,"threads_started":1}
I20260806 17:05:12.095099 24743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76edeb977b3e40cfa5f89788dac0c248" candidate_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac10b9995a6403a8d29aa601ab87e63" is_pre_election: true
I20260806 17:05:12.095489 24743 raft_consensus.cc:2454] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3163e7d6f71e4214a8b8398f89a05a2e in term 0.
I20260806 17:05:12.095546 24656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:12.089725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:60565 (local address 127.9.111.65:33917)
0806 17:05:12.089887 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:12.089892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:12.090204 (+ 312us) server_negotiation.cc:407] Connection header received
0806 17:05:12.090314 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:12.090317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:12.090370 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:12.090491 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:12.090789 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.091457 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.093055 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.093990 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.094123 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:12.095254 (+ 1131us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:12.095305 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 17:05:12.095418 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 17:05:12.096057 24812 leader_election.cc:302] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3163e7d6f71e4214a8b8398f89a05a2e, 9ac10b9995a6403a8d29aa601ab87e63; no voters:
I20260806 17:05:12.096134 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76edeb977b3e40cfa5f89788dac0c248" candidate_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a43e12e8e3464576b6427599e1be6926" is_pre_election: true
I20260806 17:05:12.096529 24601 raft_consensus.cc:2454] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3163e7d6f71e4214a8b8398f89a05a2e in term 0.
I20260806 17:05:12.096549 24948 raft_consensus.cc:2781] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:05:12.096640 24948 raft_consensus.cc:482] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:05:12.096710 24948 raft_consensus.cc:3037] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:12.098466 24948 raft_consensus.cc:504] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } }
I20260806 17:05:12.098894 24948 leader_election.cc:288] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [CANDIDATE]: Term 1 election: Requested vote from peers 9ac10b9995a6403a8d29aa601ab87e63 (127.9.111.66:36489), a43e12e8e3464576b6427599e1be6926 (127.9.111.65:33917)
I20260806 17:05:12.099426 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76edeb977b3e40cfa5f89788dac0c248" candidate_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a43e12e8e3464576b6427599e1be6926"
I20260806 17:05:12.099480 24743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76edeb977b3e40cfa5f89788dac0c248" candidate_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac10b9995a6403a8d29aa601ab87e63"
I20260806 17:05:12.099581 24601 raft_consensus.cc:3037] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:12.099653 24743 raft_consensus.cc:3037] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:12.100709 24601 raft_consensus.cc:2454] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3163e7d6f71e4214a8b8398f89a05a2e in term 1.
I20260806 17:05:12.100803 24743 raft_consensus.cc:2454] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3163e7d6f71e4214a8b8398f89a05a2e in term 1.
I20260806 17:05:12.101135 24812 leader_election.cc:302] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3163e7d6f71e4214a8b8398f89a05a2e, a43e12e8e3464576b6427599e1be6926; no voters:
I20260806 17:05:12.101341 24948 raft_consensus.cc:2781] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:05:12.101670 24948 raft_consensus.cc:686] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [term 1 LEADER]: Becoming Leader. State: Replica: 3163e7d6f71e4214a8b8398f89a05a2e, State: Running, Role: LEADER
I20260806 17:05:12.102102 24948 consensus_queue.cc:237] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } }
I20260806 17:05:12.106757 24451 catalog_manager.cc:5495] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3163e7d6f71e4214a8b8398f89a05a2e (127.9.111.67). New cstate: current_term: 1 leader_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } health_report { overall_health: UNKNOWN } } }
I20260806 17:05:12.121902 24940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:12.117292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.9.111.67:33181)
0806 17:05:12.117445 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:12.117450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:12.117465 (+ 15us) server_negotiation.cc:407] Connection header received
0806 17:05:12.117632 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:12.117635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:12.117692 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:12.117920 (+ 228us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:05:12.118382 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.118957 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.120092 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:12.120426 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:12.120544 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:12.120943 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:05:12.121427 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:05:12.121569 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:12.121637 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 17:05:12.121710 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 17:05:12.130023 24743 raft_consensus.cc:1261] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63 [term 1 FOLLOWER]: Refusing update from remote peer 3163e7d6f71e4214a8b8398f89a05a2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:05:12.129791 24601 raft_consensus.cc:1261] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Refusing update from remote peer 3163e7d6f71e4214a8b8398f89a05a2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:05:12.131266 24950 consensus_queue.cc:1035] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:05:12.132022 24948 consensus_queue.cc:1035] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:05:12.139377 24954 mvcc.cc:205] Tried to move back new op lower bound from 7315603096074915840 to 7315603095973261312. Current Snapshot: MvccSnapshot[applied={T|T < 7315603096074915840}]
I20260806 17:05:12.145098 24955 mvcc.cc:205] Tried to move back new op lower bound from 7315603096074915840 to 7315603095973261312. Current Snapshot: MvccSnapshot[applied={T|T < 7315603096074915840}]
I20260806 17:05:12.151172 24449 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:05:12.151376 24449 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:05:12.154552 24449 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec: Sending DeleteTablet for 3 replicas of tablet 76edeb977b3e40cfa5f89788dac0c248
I20260806 17:05:12.155676 24581 tablet_service.cc:1514] Processing DeleteTablet for tablet 76edeb977b3e40cfa5f89788dac0c248 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:05:12 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:33640
I20260806 17:05:12.155691 24860 tablet_service.cc:1514] Processing DeleteTablet for tablet 76edeb977b3e40cfa5f89788dac0c248 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:05:12 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:37810
I20260806 17:05:12.155706 24723 tablet_service.cc:1514] Processing DeleteTablet for tablet 76edeb977b3e40cfa5f89788dac0c248 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:05:12 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:32792
I20260806 17:05:12.156417 24965 tablet_replica.cc:330] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63: stopping tablet replica
I20260806 17:05:12.156544 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 24493
I20260806 17:05:12.156675 24964 tablet_replica.cc:330] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e: stopping tablet replica
I20260806 17:05:12.156920 24965 raft_consensus.cc:2229] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:05:12.157117 24964 raft_consensus.cc:2229] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:05:12.157342 24965 raft_consensus.cc:2258] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:05:12.157673 24964 raft_consensus.cc:2258] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:05:12.159515 24965 ts_tablet_manager.cc:1905] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:05:12.159515 24964 ts_tablet_manager.cc:1905] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:05:12.163013 24964 ts_tablet_manager.cc:1918] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:05:12.163013 24965 ts_tablet_manager.cc:1918] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:05:12.163362 24965 log.cc:1196] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/76edeb977b3e40cfa5f89788dac0c248
I20260806 17:05:12.163606 24964 log.cc:1196] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/76edeb977b3e40cfa5f89788dac0c248
I20260806 17:05:12.164018 24965 ts_tablet_manager.cc:1939] T 76edeb977b3e40cfa5f89788dac0c248 P 9ac10b9995a6403a8d29aa601ab87e63: Deleting consensus metadata
I20260806 17:05:12.164409 24964 ts_tablet_manager.cc:1939] T 76edeb977b3e40cfa5f89788dac0c248 P 3163e7d6f71e4214a8b8398f89a05a2e: Deleting consensus metadata
I20260806 17:05:12.165926 24437 catalog_manager.cc:4841] TS 3163e7d6f71e4214a8b8398f89a05a2e (127.9.111.67:33181): tablet 76edeb977b3e40cfa5f89788dac0c248 (table test-table [id=9381837fc4a5461184208a7d562e962e]) successfully deleted
I20260806 17:05:12.165926 24438 catalog_manager.cc:4841] TS 9ac10b9995a6403a8d29aa601ab87e63 (127.9.111.66:36489): tablet 76edeb977b3e40cfa5f89788dac0c248 (table test-table [id=9381837fc4a5461184208a7d562e962e]) successfully deleted
W20260806 17:05:12.172260 24438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.111.65:33917 (error 108)
I20260806 17:05:12.172982 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 24652
I20260806 17:05:12.176126 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:12.175723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:33917 (local address 127.0.0.1:33650)
0806 17:05:12.175884 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:12.176028 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.65:33917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 17:05:12.176410 24438 catalog_manager.cc:4584] TS a43e12e8e3464576b6427599e1be6926 (127.9.111.65:33917): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 76edeb977b3e40cfa5f89788dac0c248: Network error: Client connection negotiation failed: client connection to 127.9.111.65:33917: connect: Connection refused (error 111)
I20260806 17:05:12.182073 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 24794
I20260806 17:05:12.188536 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 24420
I20260806 17:05:12.196605 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:37601
--webserver_interface=127.9.111.126
--webserver_port=33425
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:37601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:05:12.328934 24967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:12.329346 24967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:12.329444 24967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:12.334478 24967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:05:12.334626 24967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:12.334666 24967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:12.334703 24967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:05:12.334743 24967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:05:12.341360 24967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:37601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:37601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=33425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:12.342700 24967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:12.343888 24967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:12.355432 24973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:12.355428 24972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:12.355430 24975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:12.355937 24967 server_base.cc:1034] running on GCE node
I20260806 17:05:12.356616 24967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:12.358059 24967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:12.359375 24967 hybrid_clock.cc:648] HybridClock initialized: now 1786035912359357 us; error 46 us; skew 500 ppm
Aug 06 17:05:12 dist-test-slave-69rg krb5kdc[23848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035912, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:12.362890 24967 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:05:12.363999 24967 webserver.cc:460] Webserver started at http://127.9.111.126:33425/ using document root <none> and password file <none>
I20260806 17:05:12.364635 24967 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:12.364741 24967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:12.370003 24967 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 17:05:12.372746 24982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:12.374532 24967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:05:12.374724 24967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "d28bb840247d47e3a039d2885030f6ec"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "555b15fbde0b497b35a76cb59ed6b938"
server_key_iv: "5a8e6ca5ef3df8c8e076fc8039f15ba3"
server_key_version: "encryptionkey@0"
I20260806 17:05:12.375633 24967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:12.387200 24967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:12.391476 24967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:12.391739 24967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:12.401185 24967 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:37601
I20260806 17:05:12.401496 25036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:37601 every 8 connection(s)
I20260806 17:05:12.402585 24967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:05:12.408473 25037 sys_catalog.cc:263] Verifying existing consensus state
I20260806 17:05:12.410612 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 24967
I20260806 17:05:12.410930 25037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec: Bootstrap starting.
I20260806 17:05:12.411209 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:33917
--local_ip_for_outbound_sockets=127.9.111.65
--tserver_master_addrs=127.9.111.126:37601
--webserver_port=33659
--webserver_interface=127.9.111.65
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:05:12.431499 25037 log.cc:824] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:12.444854 25037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 17:05:12.445382 25037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec: Bootstrap complete.
I20260806 17:05:12.451637 25037 raft_consensus.cc:348] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28bb840247d47e3a039d2885030f6ec" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37601 } }
I20260806 17:05:12.452265 25037 raft_consensus.cc:729] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d28bb840247d47e3a039d2885030f6ec, State: Initialized, Role: FOLLOWER
I20260806 17:05:12.453032 25037 consensus_queue.cc:260] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28bb840247d47e3a039d2885030f6ec" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37601 } }
I20260806 17:05:12.453289 25037 raft_consensus.cc:388] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:05:12.453393 25037 raft_consensus.cc:482] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:05:12.453522 25037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 2 FOLLOWER]: Advancing to term 3
I20260806 17:05:12.455992 25037 raft_consensus.cc:504] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28bb840247d47e3a039d2885030f6ec" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37601 } }
I20260806 17:05:12.456651 25037 leader_election.cc:302] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d28bb840247d47e3a039d2885030f6ec; no voters:
I20260806 17:05:12.457119 25037 leader_election.cc:288] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 17:05:12.457238 25041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 3 FOLLOWER]: Leader election won for term 3
I20260806 17:05:12.457459 25041 raft_consensus.cc:686] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [term 3 LEADER]: Becoming Leader. State: Replica: d28bb840247d47e3a039d2885030f6ec, State: Running, Role: LEADER
I20260806 17:05:12.457852 25041 consensus_queue.cc:237] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28bb840247d47e3a039d2885030f6ec" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37601 } }
I20260806 17:05:12.458832 25037 sys_catalog.cc:564] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:05:12.459733 25042 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d28bb840247d47e3a039d2885030f6ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28bb840247d47e3a039d2885030f6ec" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37601 } } }
I20260806 17:05:12.459785 25043 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader d28bb840247d47e3a039d2885030f6ec. Latest consensus state: current_term: 3 leader_uuid: "d28bb840247d47e3a039d2885030f6ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28bb840247d47e3a039d2885030f6ec" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 37601 } } }
I20260806 17:05:12.459890 25042 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [sys.catalog]: This master's current role is: LEADER
I20260806 17:05:12.459915 25043 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec [sys.catalog]: This master's current role is: LEADER
I20260806 17:05:12.464046 25056 catalog_manager.cc:1259] Loaded cluster ID: ef0b6ba20412452e9e38e349a8649a15
I20260806 17:05:12.464152 25056 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec: loading cluster ID for follower catalog manager: success
W20260806 17:05:12.465138 25056 catalog_manager.cc:1576] T 00000000000000000000000000000000 P d28bb840247d47e3a039d2885030f6ec: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260806 17:05:12.465236 25056 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260806 17:05:12.465332 25050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:05:12.468467 25050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:05:12.469264 25050 catalog_manager.cc:1259] Loaded cluster ID: ef0b6ba20412452e9e38e349a8649a15
I20260806 17:05:12.469345 25050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 17:05:12.470057 25050 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f910cf559ad google::LogMessage::Fail() at ??:0
@ 0x7f910cf5888a google::LogMessage::SendToLog() at ??:0
@ 0x7f910cf554de google::LogMessage::Flush() at ??:0
@ 0x7f910cf56a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f911789afdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f911789bd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9117898579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f91178dc640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f910f1a73d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f910f1a7c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f910f1a93f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f910f198512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f91107e06db start_thread at ??:0
@ 0x7f910ba5361f clone at ??:0
W20260806 17:05:12.579607 25039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:12.579939 25039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:12.580034 25039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:12.584864 25039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:12.584983 25039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:12.585062 25039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:05:12.591853 25039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:33917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=33659
--tserver_master_addrs=127.9.111.126:37601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:12.593186 25039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:12.594377 25039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:12.606657 25082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:12.606702 25079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:12.606905 25039 server_base.cc:1034] running on GCE node
W20260806 17:05:12.607048 25080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:12.607642 25039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:12.608608 25039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:12.609861 25039 hybrid_clock.cc:648] HybridClock initialized: now 1786035912609810 us; error 65 us; skew 500 ppm
Aug 06 17:05:12 dist-test-slave-69rg krb5kdc[23848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035912, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:12.614001 25039 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:05:12.615338 25039 webserver.cc:460] Webserver started at http://127.9.111.65:33659/ using document root <none> and password file <none>
I20260806 17:05:12.616124 25039 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:12.616242 25039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:12.622094 25039 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 17:05:12.625154 25089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:12.626979 25039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:05:12.627142 25039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "a43e12e8e3464576b6427599e1be6926"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "42b3cc230022a6f433f7c4f3379a0e77"
server_key_iv: "0a27fffc785e16259c775eaaf2f320ef"
server_key_version: "encryptionkey@0"
I20260806 17:05:12.628007 25039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:12.652741 25039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:12.657245 25039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:12.657493 25039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:12.658352 25039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:05:12.659986 25096 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 17:05:12.664580 25039 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 17:05:12.664691 25039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260806 17:05:12.664767 25039 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 17:05:12.667235 25039 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 17:05:12.667330 25039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.001s
I20260806 17:05:12.667711 25096 tablet_bootstrap.cc:492] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926: Bootstrap starting.
I20260806 17:05:12.679227 25039 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:33917
I20260806 17:05:12.679945 25209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:33917 every 8 connection(s)
I20260806 17:05:12.680630 25039 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:05:12.682053 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 25039
I20260806 17:05:12.682536 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:36489
--local_ip_for_outbound_sockets=127.9.111.66
--tserver_master_addrs=127.9.111.126:37601
--webserver_port=43295
--webserver_interface=127.9.111.66
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:05:12.689977 25096 log.cc:824] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:12.700176 25096 tablet_bootstrap.cc:492] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 17:05:12.700721 25096 tablet_bootstrap.cc:492] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926: Bootstrap complete.
I20260806 17:05:12.702208 25096 ts_tablet_manager.cc:1397] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.008s
I20260806 17:05:12.705920 25096 raft_consensus.cc:348] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } }
I20260806 17:05:12.706568 25096 raft_consensus.cc:729] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43e12e8e3464576b6427599e1be6926, State: Initialized, Role: FOLLOWER
I20260806 17:05:12.707175 25096 consensus_queue.cc:260] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } }
I20260806 17:05:12.708421 25096 ts_tablet_manager.cc:1428] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260806 17:05:12.724406 25213 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:05:12.684271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:56315)
0806 17:05:12.685751 (+ 1480us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:12.685855 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:12.687187 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:12.723393 (+ 36206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1184,"thread_start_us":191,"threads_started":1}
W20260806 17:05:12.725126 25210 heartbeater.cc:644] Failed to heartbeat to 127.9.111.126:37601 (0 consecutive failures): Network error: Failed to ping master at 127.9.111.126:37601: Client connection negotiation failed: client connection to 127.9.111.126:37601: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 17:05:12.726109 25213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:12.725759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:58669)
0806 17:05:12.725938 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:12.726028 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 17:05:12.727020 25213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:12.726770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:38795)
0806 17:05:12.726897 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:12.726954 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 17:05:12.727268 25210 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 17:05:12.812307 25212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:12.812605 25212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:12.812654 25212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:12.817072 25212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:12.817173 25212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:12.817238 25212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:05:12.823801 25212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:36489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=43295
--tserver_master_addrs=127.9.111.126:37601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:12.825145 25212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:12.826574 25212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:12.838217 25224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:12.838287 25212 server_base.cc:1034] running on GCE node
W20260806 17:05:12.838218 25220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:12.838224 25221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:12.838956 25212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:12.839723 25212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:12.840924 25212 hybrid_clock.cc:648] HybridClock initialized: now 1786035912840880 us; error 67 us; skew 500 ppm
Aug 06 17:05:12 dist-test-slave-69rg krb5kdc[23848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035912, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:12.844201 25212 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:05:12.845214 25212 webserver.cc:460] Webserver started at http://127.9.111.66:43295/ using document root <none> and password file <none>
I20260806 17:05:12.845885 25212 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:12.845984 25212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:12.850261 25212 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 17:05:12.852577 25231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:12.854296 25212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:05:12.854457 25212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "9ac10b9995a6403a8d29aa601ab87e63"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "485b866cd6959ab51e78be66b17724dc"
server_key_iv: "d82ea052fac05e4e338d87ef443c2d84"
server_key_version: "encryptionkey@0"
I20260806 17:05:12.855072 25212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:12.877928 25212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:12.881518 25212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:12.881779 25212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:12.882509 25212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:05:12.883486 25212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:05:12.883563 25212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:12.883636 25212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:05:12.883682 25212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:12.893373 25212 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:36489
I20260806 17:05:12.893392 25349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:36489 every 8 connection(s)
I20260806 17:05:12.894519 25212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:05:12.898873 25352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:12.896752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:41303)
0806 17:05:12.897266 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:12.897393 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":132,"threads_started":1}
W20260806 17:05:12.899456 25350 heartbeater.cc:644] Failed to heartbeat to 127.9.111.126:37601 (0 consecutive failures): Network error: Failed to ping master at 127.9.111.126:37601: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
I20260806 17:05:12.900187 25352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:12.899935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:41097)
0806 17:05:12.900062 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:12.900126 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 17:05:12.900802 25352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:12.900608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:34521)
0806 17:05:12.900706 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:12.900748 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 17:05:12.900986 25350 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 17:05:12.902591 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 25212
I20260806 17:05:12.903010 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:33181
--local_ip_for_outbound_sockets=127.9.111.67
--tserver_master_addrs=127.9.111.126:37601
--webserver_port=45083
--webserver_interface=127.9.111.67
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:05:13.040983 25353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:13.041323 25353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:13.041373 25353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:13.047312 25353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:13.047435 25353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:13.047518 25353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:05:13.055029 25353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:33181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=45083
--tserver_master_addrs=127.9.111.126:37601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:13.056313 25353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:13.057477 25353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:13.069156 25359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:13.069205 25353 server_base.cc:1034] running on GCE node
W20260806 17:05:13.069156 25362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:13.069171 25358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:13.069990 25353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:13.070708 25353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:13.071923 25353 hybrid_clock.cc:648] HybridClock initialized: now 1786035913071900 us; error 50 us; skew 500 ppm
Aug 06 17:05:13 dist-test-slave-69rg krb5kdc[23848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035913, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:13.075651 25353 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:05:13.076717 25353 webserver.cc:460] Webserver started at http://127.9.111.67:45083/ using document root <none> and password file <none>
I20260806 17:05:13.077339 25353 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:13.077414 25353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:13.082329 25353 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:05:13.084957 25369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:13.086766 25353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:05:13.086905 25353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "3163e7d6f71e4214a8b8398f89a05a2e"
format_stamp: "Formatted at 2026-08-06 17:05:10 on dist-test-slave-69rg"
server_key: "042ed8f827281c2561abf61c87dbebf7"
server_key_iv: "d4ff289cdd3a23837a481ec7f398f6ac"
server_key_version: "encryptionkey@0"
I20260806 17:05:13.087567 25353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:13.108289 25353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:13.112149 25353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:13.112341 25353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:13.112941 25353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:05:13.113790 25353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:05:13.113897 25353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:13.113945 25353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:05:13.113966 25353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:13.124681 25353 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:33181
I20260806 17:05:13.124704 25490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:33181 every 8 connection(s)
I20260806 17:05:13.125921 25353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:05:13.130986 25493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:13.128674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.67:46935)
0806 17:05:13.129356 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:13.129518 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":169,"threads_started":1}
W20260806 17:05:13.131621 25491 heartbeater.cc:644] Failed to heartbeat to 127.9.111.126:37601 (0 consecutive failures): Network error: Failed to ping master at 127.9.111.126:37601: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
I20260806 17:05:13.132431 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 25353
I20260806 17:05:13.132470 25493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:13.132168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.67:37853)
0806 17:05:13.132314 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:13.132374 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 17:05:13.133176 25493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:13.132937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.67:57063)
0806 17:05:13.133063 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:13.133115 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 17:05:13.133400 25491 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 17:05:13.728538 25495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:13.728033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:59329)
0806 17:05:13.728365 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:13.728423 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
I20260806 17:05:13.902094 25496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:13.901584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:55253)
0806 17:05:13.901918 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:13.901983 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
I20260806 17:05:14.134619 25497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:14.134143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.67:52499)
0806 17:05:14.134442 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:14.134504 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":81,"threads_started":1}
I20260806 17:05:14.376197 25498 raft_consensus.cc:482] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:05:14.376435 25498 raft_consensus.cc:504] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } }
I20260806 17:05:14.377638 25498 leader_election.cc:288] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3163e7d6f71e4214a8b8398f89a05a2e (127.9.111.67:33181), 9ac10b9995a6403a8d29aa601ab87e63 (127.9.111.66:36489)
Aug 06 17:05:14 dist-test-slave-69rg krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035912, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.67@KRBTEST.COM
Aug 06 17:05:14 dist-test-slave-69rg krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035912, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.66@KRBTEST.COM
I20260806 17:05:14.392710 25500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:14.378018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:36489 (local address 127.9.111.65:55891)
0806 17:05:14.378390 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:14.378404 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:14.378496 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:14.380015 (+ 1519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:14.380021 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:14.380589 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:14.381060 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:14.381077 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:14.382482 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:14.382486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:14.383471 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:14.383481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:14.384189 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:14.384193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:14.384409 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:14.385117 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:14.385129 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:14.387039 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:14.389472 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:14.389479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:14.389494 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:14.390027 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:14.390512 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:14.390516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:14.390518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:14.390752 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:14.391610 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:14.391617 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:14.391919 (+ 302us) client_negotiation.cc:769] Sending connection context
0806 17:05:14.392120 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 17:05:14.392493 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":150,"spinlock_wait_cycles":33536,"thread_start_us":84,"threads_started":1}
I20260806 17:05:14.392731 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:14.378058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:39677 (local address 127.9.111.67:33181)
0806 17:05:14.378737 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:14.378756 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:14.378789 (+ 33us) server_negotiation.cc:407] Connection header received
0806 17:05:14.379438 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:14.379443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:14.379732 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:14.380011 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:14.381363 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:14.382415 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:14.383598 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:14.384062 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:14.384231 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:14.387056 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:14.387061 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:14.387074 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:14.387123 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:14.389317 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:14.390202 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:14.390209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:14.390218 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:14.390391 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:14.390903 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:14.390907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:14.390908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:14.391473 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:14.391616 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:14.392286 (+ 670us) server_negotiation.cc:299] Negotiation successful
0806 17:05:14.392591 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":112,"threads_started":1}
I20260806 17:05:14.392937 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:14.377859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:33181 (local address 127.9.111.65:39677)
0806 17:05:14.378367 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:14.378384 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:14.378525 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:14.380015 (+ 1490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:14.380021 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:14.380589 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:14.381062 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:14.381079 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:14.382543 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:14.382547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:14.383476 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:14.383486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:14.384198 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:14.384202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:14.384404 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:14.385062 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:14.385079 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:14.386821 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:14.389472 (+ 2651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:14.389479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:14.389494 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:14.390027 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:14.390512 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:14.390516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:14.390518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:14.390752 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:14.391893 (+ 1141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:14.391896 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:14.391970 (+ 74us) client_negotiation.cc:769] Sending connection context
0806 17:05:14.392120 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 17:05:14.392636 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":116,"spinlock_wait_cycles":36992,"thread_start_us":64,"threads_started":1}
I20260806 17:05:14.393603 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:14.378170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:55891 (local address 127.9.111.66:36489)
0806 17:05:14.378515 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:14.378535 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:14.378574 (+ 39us) server_negotiation.cc:407] Connection header received
0806 17:05:14.379284 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:14.379290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:14.379632 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:14.379946 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:14.381243 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:14.382344 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:14.383597 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:14.384064 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:14.384241 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:14.387278 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:14.387286 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:14.387299 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:14.387337 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:14.389317 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:14.390218 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:14.390223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:14.390227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:14.390391 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:14.390883 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:14.390888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:14.390890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:14.391474 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:14.391616 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:14.393090 (+ 1474us) server_negotiation.cc:299] Negotiation successful
0806 17:05:14.393405 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":111,"threads_started":1}
W20260806 17:05:14.396016 25090 leader_election.cc:341] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3163e7d6f71e4214a8b8398f89a05a2e (127.9.111.67:33181): Not found: Tablet not found: 76edeb977b3e40cfa5f89788dac0c248
W20260806 17:05:14.396675 25091 leader_election.cc:341] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ac10b9995a6403a8d29aa601ab87e63 (127.9.111.66:36489): Not found: Tablet not found: 76edeb977b3e40cfa5f89788dac0c248
I20260806 17:05:14.396754 25091 leader_election.cc:302] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a43e12e8e3464576b6427599e1be6926; no voters: 3163e7d6f71e4214a8b8398f89a05a2e, 9ac10b9995a6403a8d29aa601ab87e63
I20260806 17:05:14.397001 25498 raft_consensus.cc:2726] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 17:05:14.729781 25500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:14.729431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:44763)
0806 17:05:14.729618 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:14.729687 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 17:05:14.903549 25504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:14.902982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:34945)
0806 17:05:14.903354 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:14.903436 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":71,"threads_started":1}
I20260806 17:05:15.136301 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:15.135784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.67:43023)
0806 17:05:15.136098 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:15.136178 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":96,"threads_started":1}
I20260806 17:05:15.731241 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:15.730748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:44079)
0806 17:05:15.731060 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:15.731135 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":103,"threads_started":1}
I20260806 17:05:15.904963 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:15.904476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:43655)
0806 17:05:15.904788 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:15.904862 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":89,"threads_started":1}
I20260806 17:05:16.137717 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:16.137159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.67:60781)
0806 17:05:16.137534 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:16.137596 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":102,"threads_started":1}
I20260806 17:05:16.526918 25509 raft_consensus.cc:482] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:05:16.527068 25509 raft_consensus.cc:504] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } }
I20260806 17:05:16.527534 25509 leader_election.cc:288] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3163e7d6f71e4214a8b8398f89a05a2e (127.9.111.67:33181), 9ac10b9995a6403a8d29aa601ab87e63 (127.9.111.66:36489)
W20260806 17:05:16.528487 25091 leader_election.cc:341] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ac10b9995a6403a8d29aa601ab87e63 (127.9.111.66:36489): Not found: Tablet not found: 76edeb977b3e40cfa5f89788dac0c248
W20260806 17:05:16.528587 25090 leader_election.cc:341] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3163e7d6f71e4214a8b8398f89a05a2e (127.9.111.67:33181): Not found: Tablet not found: 76edeb977b3e40cfa5f89788dac0c248
I20260806 17:05:16.528728 25090 leader_election.cc:302] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a43e12e8e3464576b6427599e1be6926; no voters: 3163e7d6f71e4214a8b8398f89a05a2e, 9ac10b9995a6403a8d29aa601ab87e63
I20260806 17:05:16.528903 25509 raft_consensus.cc:2726] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 17:05:16.732522 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:16.732062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:60651)
0806 17:05:16.732356 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:16.732420 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":81,"threads_started":1}
I20260806 17:05:16.906431 25511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:16.905965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:59887)
0806 17:05:16.906251 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:16.906331 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":96,"threads_started":1}
I20260806 17:05:17.139117 25512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:17.138585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.67:49529)
0806 17:05:17.138952 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:17.139016 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":92,"threads_started":1}
I20260806 17:05:17.734058 25513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:17.733485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:51633)
0806 17:05:17.733901 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:17.733963 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":105,"threads_started":1}
I20260806 17:05:17.907871 25514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:17.907337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:46901)
0806 17:05:17.907649 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:17.907714 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":117,"threads_started":1}
I20260806 17:05:18.140456 25515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:18.139995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.67:41605)
0806 17:05:18.140289 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:18.140355 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
I20260806 17:05:18.630650 25516 raft_consensus.cc:482] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:05:18.630780 25516 raft_consensus.cc:504] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } }
I20260806 17:05:18.631234 25516 leader_election.cc:288] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3163e7d6f71e4214a8b8398f89a05a2e (127.9.111.67:33181), 9ac10b9995a6403a8d29aa601ab87e63 (127.9.111.66:36489)
W20260806 17:05:18.632212 25090 leader_election.cc:341] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3163e7d6f71e4214a8b8398f89a05a2e (127.9.111.67:33181): Not found: Tablet not found: 76edeb977b3e40cfa5f89788dac0c248
W20260806 17:05:18.632313 25091 leader_election.cc:341] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ac10b9995a6403a8d29aa601ab87e63 (127.9.111.66:36489): Not found: Tablet not found: 76edeb977b3e40cfa5f89788dac0c248
I20260806 17:05:18.632359 25090 leader_election.cc:302] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a43e12e8e3464576b6427599e1be6926; no voters: 3163e7d6f71e4214a8b8398f89a05a2e, 9ac10b9995a6403a8d29aa601ab87e63
I20260806 17:05:18.632519 25516 raft_consensus.cc:2726] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 17:05:18.735448 25517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:18.734903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:56667)
0806 17:05:18.735277 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:18.735340 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":114,"threads_started":1}
I20260806 17:05:18.909219 25518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:18.908731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:42415)
0806 17:05:18.909035 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:18.909098 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":98,"threads_started":1}
I20260806 17:05:19.141673 25519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:19.141270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.67:48263)
0806 17:05:19.141521 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:19.141575 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260806 17:05:19.736691 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:19.736249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:58757)
0806 17:05:19.736529 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:19.736592 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":84,"threads_started":1}
I20260806 17:05:19.910590 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:19.910105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:33685)
0806 17:05:19.910433 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:19.910492 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":85,"threads_started":1}
I20260806 17:05:20.142985 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:20.142523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.67:42537)
0806 17:05:20.142824 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:20.142889 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":77,"threads_started":1}
I20260806 17:05:20.738008 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:20.737491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:34515)
0806 17:05:20.737778 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:20.737909 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":89,"threads_started":1}
I20260806 17:05:20.912108 25524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:20.911562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:43499)
0806 17:05:20.911921 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:20.912003 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
I20260806 17:05:21.144320 25525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:21.143863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.67:46211)
0806 17:05:21.144136 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:21.144202 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260806 17:05:21.164191 25526 raft_consensus.cc:482] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:05:21.164359 25526 raft_consensus.cc:504] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3163e7d6f71e4214a8b8398f89a05a2e" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 33181 } } peers { permanent_uuid: "9ac10b9995a6403a8d29aa601ab87e63" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36489 } } peers { permanent_uuid: "a43e12e8e3464576b6427599e1be6926" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 33917 } }
I20260806 17:05:21.164894 25526 leader_election.cc:288] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3163e7d6f71e4214a8b8398f89a05a2e (127.9.111.67:33181), 9ac10b9995a6403a8d29aa601ab87e63 (127.9.111.66:36489)
W20260806 17:05:21.165830 25091 leader_election.cc:341] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ac10b9995a6403a8d29aa601ab87e63 (127.9.111.66:36489): Not found: Tablet not found: 76edeb977b3e40cfa5f89788dac0c248
W20260806 17:05:21.165951 25090 leader_election.cc:341] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3163e7d6f71e4214a8b8398f89a05a2e (127.9.111.67:33181): Not found: Tablet not found: 76edeb977b3e40cfa5f89788dac0c248
I20260806 17:05:21.166009 25090 leader_election.cc:302] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a43e12e8e3464576b6427599e1be6926; no voters: 3163e7d6f71e4214a8b8398f89a05a2e, 9ac10b9995a6403a8d29aa601ab87e63
I20260806 17:05:21.166184 25526 raft_consensus.cc:2726] T 76edeb977b3e40cfa5f89788dac0c248 P a43e12e8e3464576b6427599e1be6926 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 17:05:21.739310 25527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:21.738830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:48091)
0806 17:05:21.739143 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:21.739207 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260806 17:05:21.913599 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:21.913095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:44029)
0806 17:05:21.913422 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:21.913491 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":127,"threads_started":1}
I20260806 17:05:22.145800 25529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:22.145325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.67:43309)
0806 17:05:22.145632 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:22.145701 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":93,"threads_started":1}
I20260806 17:05:22.741083 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:22.740525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:60117)
0806 17:05:22.740890 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:22.740972 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":115,"threads_started":1}
I20260806 17:05:22.915022 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:22.914572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:46291)
0806 17:05:22.914861 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:22.914922 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":77,"threads_started":1}
I20260806 17:05:23.147360 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:23.146829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.67:46281)
0806 17:05:23.147194 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:23.147260 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":89,"threads_started":1}
I20260806 17:05:23.742555 25533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:23.742082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:39891)
0806 17:05:23.742397 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:23.742455 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":89,"threads_started":1}
I20260806 17:05:23.916525 25534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:23.916038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:47387)
0806 17:05:23.916337 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:23.916416 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":121,"threads_started":1}
I20260806 17:05:24.148893 25535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:24.148348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.67:39443)
0806 17:05:24.148702 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:24.148775 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":107,"threads_started":1}
I20260806 17:05:24.744141 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:24.743626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:42969)
0806 17:05:24.743964 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:24.744038 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":113,"threads_started":1}
I20260806 17:05:24.918020 25537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:24.917450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:43439)
0806 17:05:24.917780 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:24.917917 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":108,"threads_started":1}
I20260806 17:05:25.150583 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:25.150042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.67:60795)
0806 17:05:25.150384 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:25.150457 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":85,"threads_started":1}
I20260806 17:05:25.745612 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:25.745068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:59063)
0806 17:05:25.745397 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:25.745458 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":124,"threads_started":1}
I20260806 17:05:25.919536 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:25.919029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:33813)
0806 17:05:25.919361 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:25.919430 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":96,"threads_started":1}
I20260806 17:05:26.151989 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:26.151501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.67:36597)
0806 17:05:26.151822 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:26.151887 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":98,"threads_started":1}
I20260806 17:05:26.747103 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:26.746594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:39355)
0806 17:05:26.746931 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:26.746998 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":98,"threads_started":1}
I20260806 17:05:26.920897 25543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:26.920448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:52957)
0806 17:05:26.920733 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:26.920795 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":111,"threads_started":1}
I20260806 17:05:27.153276 25544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:27.152834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.67:34997)
0806 17:05:27.153114 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:27.153177 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260806 17:05:27.748710 25545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:27.748253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.65:50699)
0806 17:05:27.748534 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:27.748596 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":117,"threads_started":1}
I20260806 17:05:27.922576 25546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:27.922086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:37601 (local address 127.9.111.66:33693)
0806 17:05:27.922405 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:27.922467 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:37601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":90,"threads_started":1}
I20260806 17:05:28.134927 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 25039
I20260806 17:05:28.141860 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 25212
I20260806 17:05:28.148296 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 25353
2026-08-06T17:05:28Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19452 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:05:28 dist-test-slave-69rg krb5kdc[25549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:05:28 dist-test-slave-69rg krb5kdc[25549](info): set up 2 sockets
Aug 06 17:05:28 dist-test-slave-69rg krb5kdc[25549](info): commencing operation
krb5kdc: starting...
W20260806 17:05:29.258308 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:05:29 dist-test-slave-69rg krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:05:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:05:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:05:29.398658 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:34923
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:34923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:05:29.515756 25563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:29.516047 25563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:29.516088 25563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:29.520177 25563 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 17:05:29.520233 25563 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 17:05:29.520249 25563 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 17:05:29.520263 25563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:05:29.520282 25563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:29.520300 25563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:29.520362 25563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:05:29.520388 25563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:05:29.525774 25563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:34923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:34923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:29.526952 25563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:29.528024 25563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:29.536913 25573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:29.536901 25569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:29.537009 25563 server_base.cc:1034] running on GCE node
W20260806 17:05:29.536901 25570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:29.537801 25563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:29.538996 25563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:29.540191 25563 hybrid_clock.cc:648] HybridClock initialized: now 1786035929540173 us; error 33 us; skew 500 ppm
Aug 06 17:05:29 dist-test-slave-69rg krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035929, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:29.547787 25563 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:05:29.548902 25563 webserver.cc:460] Webserver started at http://127.9.111.126:36375/ using document root <none> and password file <none>
I20260806 17:05:29.549587 25563 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:29.549693 25563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:29.550005 25563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:05:29.552219 25563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "d0be2ea560f14f1aa80e061efccb4210"
format_stamp: "Formatted at 2026-08-06 17:05:29 on dist-test-slave-69rg"
server_key: "ea161d1633027cfdef2dac78d80c4be7"
server_key_iv: "a8778cae1d1953c752daa2a268f83fa6"
server_key_version: "encryptionkey@0"
I20260806 17:05:29.552826 25563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "d0be2ea560f14f1aa80e061efccb4210"
format_stamp: "Formatted at 2026-08-06 17:05:29 on dist-test-slave-69rg"
server_key: "ea161d1633027cfdef2dac78d80c4be7"
server_key_iv: "a8778cae1d1953c752daa2a268f83fa6"
server_key_version: "encryptionkey@0"
I20260806 17:05:29.556901 25563 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 17:05:29.559537 25580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:29.560874 25563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 17:05:29.561041 25563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "d0be2ea560f14f1aa80e061efccb4210"
format_stamp: "Formatted at 2026-08-06 17:05:29 on dist-test-slave-69rg"
server_key: "ea161d1633027cfdef2dac78d80c4be7"
server_key_iv: "a8778cae1d1953c752daa2a268f83fa6"
server_key_version: "encryptionkey@0"
I20260806 17:05:29.561163 25563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:29.589470 25563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:29.592995 25563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:29.593220 25563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:29.600848 25563 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:34923
I20260806 17:05:29.601153 25634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:34923 every 8 connection(s)
I20260806 17:05:29.601902 25563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:05:29.604782 25635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:29.606863 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 25563
I20260806 17:05:29.607041 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:05:29.607398 9661 external_mini_cluster.cc:1428] Setting key c03c373c192856d7c5078652f22661cd
I20260806 17:05:29.611161 25635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: Bootstrap starting.
I20260806 17:05:29.613402 25635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:29.614238 25635 log.cc:824] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:29.616230 25635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: No bootstrap required, opened a new log
Aug 06 17:05:29 dist-test-slave-69rg krb5kdc[25549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:05:29.619894 25635 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0be2ea560f14f1aa80e061efccb4210" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34923 } }
I20260806 17:05:29.620227 25635 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:29.620301 25635 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0be2ea560f14f1aa80e061efccb4210, State: Initialized, Role: FOLLOWER
I20260806 17:05:29.620810 25635 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0be2ea560f14f1aa80e061efccb4210" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34923 } }
I20260806 17:05:29.620966 25635 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:05:29.621032 25635 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:05:29.621158 25635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:29.622167 25635 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0be2ea560f14f1aa80e061efccb4210" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34923 } }
I20260806 17:05:29.622515 25635 leader_election.cc:302] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0be2ea560f14f1aa80e061efccb4210; no voters:
I20260806 17:05:29.622846 25635 leader_election.cc:288] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:05:29.622928 25640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:05:29.623152 25640 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 1 LEADER]: Becoming Leader. State: Replica: d0be2ea560f14f1aa80e061efccb4210, State: Running, Role: LEADER
I20260806 17:05:29.623302 25638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:29.608855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47886 (local address 127.9.111.126:34923)
0806 17:05:29.609405 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:29.609423 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:29.609461 (+ 38us) server_negotiation.cc:407] Connection header received
0806 17:05:29.610397 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:29.610418 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:29.610772 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:29.611120 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:29.611972 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:29.612961 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:29.613924 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:29.614503 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:29.614678 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:29.617494 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:29.617499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:29.617513 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:29.617539 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:29.619861 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:29.620377 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:29.620386 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:29.620394 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:29.620526 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:29.620840 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:29.620844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:29.620846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:29.621389 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:29.621591 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:29.621953 (+ 362us) server_negotiation.cc:299] Negotiation successful
0806 17:05:29.622285 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":136,"threads_started":1}
I20260806 17:05:29.624156 25635 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:05:29.623505 25640 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0be2ea560f14f1aa80e061efccb4210" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34923 } }
I20260806 17:05:29.626780 25641 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0be2ea560f14f1aa80e061efccb4210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0be2ea560f14f1aa80e061efccb4210" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34923 } } }
I20260806 17:05:29.626818 25642 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0be2ea560f14f1aa80e061efccb4210. Latest consensus state: current_term: 1 leader_uuid: "d0be2ea560f14f1aa80e061efccb4210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0be2ea560f14f1aa80e061efccb4210" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34923 } } }
I20260806 17:05:29.626917 25641 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [sys.catalog]: This master's current role is: LEADER
I20260806 17:05:29.626922 25642 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [sys.catalog]: This master's current role is: LEADER
W20260806 17:05:29.629894 25654 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:05:29.629971 25654 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:05:29.630050 25655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:05:29.630792 25655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:05:29.636283 25655 catalog_manager.cc:1347] Generated new cluster ID: 347e20fb9f974d8da40d92dd865491af
I20260806 17:05:29.636360 25655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:05:29.650179 25655 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:05:29.651690 25655 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:05:29.660849 25655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: Generated new TSK 0
I20260806 17:05:29.661904 25655 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
I20260806 17:05:29.694097 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:34923
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:05:29.806787 25662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:29.807092 25662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:29.807178 25662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:29.811208 25662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:29.811297 25662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:29.811389 25662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:05:29.817236 25662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:34923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:29.818563 25662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:29.819669 25662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:29.829885 25667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:29.829975 25662 server_base.cc:1034] running on GCE node
W20260806 17:05:29.829983 25668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:29.829877 25670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:29.830608 25662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:29.831380 25662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:29.832600 25662 hybrid_clock.cc:648] HybridClock initialized: now 1786035929832550 us; error 54 us; skew 500 ppm
Aug 06 17:05:29 dist-test-slave-69rg krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035929, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:29.835609 25662 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:05:29.836508 25662 webserver.cc:460] Webserver started at http://127.9.111.65:41903/ using document root <none> and password file <none>
I20260806 17:05:29.837092 25662 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:29.837186 25662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:29.837432 25662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:05:29.839597 25662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "82a0d759729741da91b829b16df9b588"
format_stamp: "Formatted at 2026-08-06 17:05:29 on dist-test-slave-69rg"
server_key: "2763d4b4003a971e36d45e91173aa8d2"
server_key_iv: "486409e1827fc9090e6bb449e448deee"
server_key_version: "encryptionkey@0"
I20260806 17:05:29.840188 25662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "82a0d759729741da91b829b16df9b588"
format_stamp: "Formatted at 2026-08-06 17:05:29 on dist-test-slave-69rg"
server_key: "2763d4b4003a971e36d45e91173aa8d2"
server_key_iv: "486409e1827fc9090e6bb449e448deee"
server_key_version: "encryptionkey@0"
I20260806 17:05:29.844087 25662 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 17:05:29.846725 25677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:29.848017 25662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:05:29.848169 25662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "82a0d759729741da91b829b16df9b588"
format_stamp: "Formatted at 2026-08-06 17:05:29 on dist-test-slave-69rg"
server_key: "2763d4b4003a971e36d45e91173aa8d2"
server_key_iv: "486409e1827fc9090e6bb449e448deee"
server_key_version: "encryptionkey@0"
I20260806 17:05:29.848292 25662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:29.862900 25662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:29.866166 25662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:29.866384 25662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:29.866993 25662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:05:29.867841 25662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:05:29.867914 25662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:29.867986 25662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:05:29.868036 25662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:29.877686 25662 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:40329
I20260806 17:05:29.877708 25796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:40329 every 8 connection(s)
I20260806 17:05:29.878662 25662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:05:29.881692 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 25662
I20260806 17:05:29.881855 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:05:29.882171 9661 external_mini_cluster.cc:1428] Setting key 0d49fe9e2a10bd341cfe74bb3d1082f8
Aug 06 17:05:29 dist-test-slave-69rg krb5kdc[25549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035929, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.111.66@KRBTEST.COM; defaulting to no policy
I20260806 17:05:29.893275 25638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:29.880508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:52575 (local address 127.9.111.126:34923)
0806 17:05:29.880647 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:29.880658 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:29.881615 (+ 957us) server_negotiation.cc:407] Connection header received
0806 17:05:29.882654 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:29.882658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:29.882721 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:29.882800 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:29.883958 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:29.884506 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:29.885892 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:29.886125 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:29.886212 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:29.889246 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:29.889249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:29.889251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:29.889284 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:29.890805 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:29.891352 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:29.891356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:29.891358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:29.891446 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:29.891819 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:29.891823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:29.891824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:29.892024 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:29.892135 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:29.892850 (+ 715us) server_negotiation.cc:299] Negotiation successful
0806 17:05:29.893010 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 17:05:29.894276 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:29.880814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:52575)
0806 17:05:29.881463 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:29.881504 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:29.882448 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:29.882965 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:29.882972 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:29.883326 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:29.883814 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:29.883828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:29.884651 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:29.884655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:29.885699 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:29.885713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:29.886295 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:29.886301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:29.886542 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:29.887220 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:29.887246 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:29.889047 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:29.890938 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:29.890945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:29.890961 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:29.891238 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:29.891542 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:29.891547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:29.891549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:29.891700 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:29.892134 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:29.892142 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:29.892501 (+ 359us) client_negotiation.cc:769] Sending connection context
0806 17:05:29.892757 (+ 256us) client_negotiation.cc:240] Negotiation successful
0806 17:05:29.893143 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":114,"threads_started":1}
I20260806 17:05:29.895617 25797 heartbeater.cc:347] Connected to a master server at 127.9.111.126:34923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.66@KRBTEST.COM" created.
I20260806 17:05:29.895978 25797 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:29.896507 25797 heartbeater.cc:511] Master 127.9.111.126:34923 requested a full tablet report, sending...
I20260806 17:05:29.898381 25598 ts_manager.cc:194] Registered new tserver with Master: 82a0d759729741da91b829b16df9b588 (127.9.111.65:40329)
I20260806 17:05:29.899782 25598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:52575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
I20260806 17:05:29.913589 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:34923
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:05:30.028652 25804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:30.029003 25804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:30.029085 25804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:30.033504 25804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:30.033592 25804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:30.033725 25804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:05:30.039707 25804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:34923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:30.040783 25804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:30.041798 25804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:30.051591 25813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:30.051591 25810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:30.051748 25804 server_base.cc:1034] running on GCE node
W20260806 17:05:30.051975 25811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:30.052443 25804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:30.053047 25804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:30.054215 25804 hybrid_clock.cc:648] HybridClock initialized: now 1786035930054195 us; error 57 us; skew 500 ppm
Aug 06 17:05:30 dist-test-slave-69rg krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035930, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:30.057106 25804 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:05:30.058039 25804 webserver.cc:460] Webserver started at http://127.9.111.66:33437/ using document root <none> and password file <none>
I20260806 17:05:30.058580 25804 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:30.058640 25804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:30.058806 25804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:05:30.062052 25804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "6cadb4cd97ae44f89701819e2acb73b0"
format_stamp: "Formatted at 2026-08-06 17:05:30 on dist-test-slave-69rg"
server_key: "052df19c75270527c8d4758305f62030"
server_key_iv: "642cdf39d5eacfc67dacbd5d9f503421"
server_key_version: "encryptionkey@0"
I20260806 17:05:30.062837 25804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "6cadb4cd97ae44f89701819e2acb73b0"
format_stamp: "Formatted at 2026-08-06 17:05:30 on dist-test-slave-69rg"
server_key: "052df19c75270527c8d4758305f62030"
server_key_iv: "642cdf39d5eacfc67dacbd5d9f503421"
server_key_version: "encryptionkey@0"
I20260806 17:05:30.067880 25804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 17:05:30.070564 25821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:30.071915 25804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:05:30.072052 25804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "6cadb4cd97ae44f89701819e2acb73b0"
format_stamp: "Formatted at 2026-08-06 17:05:30 on dist-test-slave-69rg"
server_key: "052df19c75270527c8d4758305f62030"
server_key_iv: "642cdf39d5eacfc67dacbd5d9f503421"
server_key_version: "encryptionkey@0"
I20260806 17:05:30.072182 25804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:30.083565 25804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:30.086915 25804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:30.087124 25804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:30.087904 25804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:05:30.088809 25804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:05:30.088886 25804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:30.088956 25804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:05:30.089010 25804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:30.098240 25804 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:34373
I20260806 17:05:30.098311 25938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:34373 every 8 connection(s)
I20260806 17:05:30.099255 25804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:05:30.101686 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 25804
I20260806 17:05:30.101854 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:05:30.102207 9661 external_mini_cluster.cc:1428] Setting key 2f07dbb65f0d2f0de2fe5fa92fdc0a1a
Aug 06 17:05:30 dist-test-slave-69rg krb5kdc[25549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035930, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.111.67@KRBTEST.COM; defaulting to no policy
I20260806 17:05:30.113912 25638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:30.101157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:38617 (local address 127.9.111.126:34923)
0806 17:05:30.101335 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:30.101339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:30.102580 (+ 1241us) server_negotiation.cc:407] Connection header received
0806 17:05:30.103703 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:30.103707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:30.103758 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:30.103827 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:30.105035 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.105584 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.106813 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.107049 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.107173 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:30.109810 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:30.109856 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:30.109862 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:30.109897 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:30.111445 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:30.111989 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:30.111992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:30.111994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:30.112082 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:30.112434 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:30.112438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:30.112440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:30.112703 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:30.112814 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:30.113509 (+ 695us) server_negotiation.cc:299] Negotiation successful
0806 17:05:30.113664 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 17:05:30.114827 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:30.101623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:38617)
0806 17:05:30.102427 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:30.102471 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:30.103472 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:30.104000 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:30.104007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:30.104380 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:30.104886 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:30.104901 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.105736 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.105739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:30.106669 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:30.106690 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.107218 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.107223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:30.107404 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:30.108039 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:30.108056 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:30.109614 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:30.111574 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:30.111580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:30.111597 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:30.111877 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:30.112185 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:30.112188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:30.112190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:30.112333 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:30.112810 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:30.112818 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:30.113188 (+ 370us) client_negotiation.cc:769] Sending connection context
0806 17:05:30.113436 (+ 248us) client_negotiation.cc:240] Negotiation successful
0806 17:05:30.113811 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":152,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.67@KRBTEST.COM" created.
I20260806 17:05:30.116218 25939 heartbeater.cc:347] Connected to a master server at 127.9.111.126:34923
I20260806 17:05:30.116523 25939 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:30.117201 25939 heartbeater.cc:511] Master 127.9.111.126:34923 requested a full tablet report, sending...
I20260806 17:05:30.118695 25598 ts_manager.cc:194] Registered new tserver with Master: 6cadb4cd97ae44f89701819e2acb73b0 (127.9.111.66:34373)
I20260806 17:05:30.119660 25598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:38617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
I20260806 17:05:30.137655 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:34923
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:05:30.256757 25944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:30.257051 25944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:30.257097 25944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:30.261441 25944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:30.261523 25944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:30.261622 25944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:05:30.267325 25944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:34923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:30.268464 25944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:30.269556 25944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:30.279995 25950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:30.279984 25952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:30.279989 25949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:30.280439 25944 server_base.cc:1034] running on GCE node
I20260806 17:05:30.280897 25944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:30.281646 25944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:30.282842 25944 hybrid_clock.cc:648] HybridClock initialized: now 1786035930282825 us; error 56 us; skew 500 ppm
Aug 06 17:05:30 dist-test-slave-69rg krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035930, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:30.285899 25944 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:05:30.286768 25944 webserver.cc:460] Webserver started at http://127.9.111.67:36881/ using document root <none> and password file <none>
I20260806 17:05:30.287325 25944 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:30.287387 25944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:30.287559 25944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:05:30.289673 25944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "da07027f963c4af49c6472087d25f700"
format_stamp: "Formatted at 2026-08-06 17:05:30 on dist-test-slave-69rg"
server_key: "9768aee8ef24d9f869aaae47f0f6ab31"
server_key_iv: "2d66d1c2e23f1362c8481c3c1834c1ed"
server_key_version: "encryptionkey@0"
I20260806 17:05:30.290341 25944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "da07027f963c4af49c6472087d25f700"
format_stamp: "Formatted at 2026-08-06 17:05:30 on dist-test-slave-69rg"
server_key: "9768aee8ef24d9f869aaae47f0f6ab31"
server_key_iv: "2d66d1c2e23f1362c8481c3c1834c1ed"
server_key_version: "encryptionkey@0"
I20260806 17:05:30.294384 25944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:05:30.296986 25959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:30.298425 25944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:05:30.298544 25944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "da07027f963c4af49c6472087d25f700"
format_stamp: "Formatted at 2026-08-06 17:05:30 on dist-test-slave-69rg"
server_key: "9768aee8ef24d9f869aaae47f0f6ab31"
server_key_iv: "2d66d1c2e23f1362c8481c3c1834c1ed"
server_key_version: "encryptionkey@0"
I20260806 17:05:30.298713 25944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:30.311708 25944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:30.315153 25944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:30.315395 25944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:30.316084 25944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:05:30.317023 25944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:05:30.317075 25944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:30.317160 25944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:05:30.317205 25944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:30.327816 25944 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:43757
I20260806 17:05:30.327834 26077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:43757 every 8 connection(s)
I20260806 17:05:30.328824 25944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:05:30.336206 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 25944
I20260806 17:05:30.336325 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:05:30.336655 9661 external_mini_cluster.cc:1428] Setting key bd4284c2c50ef3d24380846ddadc811b
Aug 06 17:05:30 dist-test-slave-69rg krb5kdc[25549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035930, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:05:30.343909 25638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:30.330870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:50307 (local address 127.9.111.126:34923)
0806 17:05:30.331055 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:30.331059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:30.332103 (+ 1044us) server_negotiation.cc:407] Connection header received
0806 17:05:30.332995 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:30.332998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:30.333048 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:30.333127 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:30.334709 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.335280 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.336602 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.336944 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.337076 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:30.340316 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:30.340320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:30.340323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:30.340353 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:30.341787 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:30.342437 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:30.342441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:30.342442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:30.342530 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:30.342885 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:30.342887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:30.342889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:30.343083 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:30.343166 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:30.343652 (+ 486us) server_negotiation.cc:299] Negotiation successful
0806 17:05:30.343774 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 17:05:30.344758 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:30.331231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.67:50307)
0806 17:05:30.331944 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:30.331987 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:30.332793 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:30.333350 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:30.333357 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:30.333759 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:30.334543 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:30.334562 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.335405 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.335408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:30.336458 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:30.336469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.337122 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.337127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:30.337334 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:30.338236 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:30.338254 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:30.340130 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:30.341962 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:30.341967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:30.341987 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:30.342338 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:30.342650 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:30.342652 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:30.342654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:30.342797 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:30.343184 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:30.343189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:30.343433 (+ 244us) client_negotiation.cc:769] Sending connection context
0806 17:05:30.343603 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 17:05:30.343895 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":122,"threads_started":1}
I20260806 17:05:30.346078 26078 heartbeater.cc:347] Connected to a master server at 127.9.111.126:34923
I20260806 17:05:30.346503 26078 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:30.347069 26078 heartbeater.cc:511] Master 127.9.111.126:34923 requested a full tablet report, sending...
I20260806 17:05:30.348299 25598 ts_manager.cc:194] Registered new tserver with Master: da07027f963c4af49c6472087d25f700 (127.9.111.67:43757)
I20260806 17:05:30.348915 25598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:50307
I20260806 17:05:30.351307 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 17:05:30.360955 25638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:30.353544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47900 (local address 127.9.111.126:34923)
0806 17:05:30.353679 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:30.353683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:30.353736 (+ 53us) server_negotiation.cc:407] Connection header received
0806 17:05:30.353905 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:30.353908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:30.353951 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:30.354027 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:30.354784 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.355422 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.356239 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.356471 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.356558 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:30.358088 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:30.358091 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:30.358093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:30.358161 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:30.359419 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:30.359822 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:30.359825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:30.359827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:30.359914 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:30.360151 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:30.360153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:30.360154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:30.360403 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:30.360514 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:30.360721 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 17:05:30.360827 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 17:05:30.364004 25598 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:05:30.367554 25598 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:05:30.382117 26089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:30.376255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:34373 (local address 127.0.0.1:35462)
0806 17:05:30.376656 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:30.376702 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:30.376834 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:30.377564 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:30.377570 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:30.377596 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:30.377746 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:30.377754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.379138 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.379142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:30.380414 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:30.380424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.381496 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.381500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:30.381611 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:30.381653 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 17:05:30.381774 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 17:05:30.381930 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":110,"threads_started":1}
I20260806 17:05:30.382143 26091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:30.376367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.9.111.66:34373)
0806 17:05:30.376835 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:30.376846 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:30.376871 (+ 25us) server_negotiation.cc:407] Connection header received
0806 17:05:30.376965 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:30.376970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:30.377183 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:30.377349 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:30.377917 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.378989 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.380551 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.381366 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.381475 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:30.381615 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:30.381858 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 17:05:30.381962 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":112,"threads_started":1}
I20260806 17:05:30.382519 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:30.376319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:40329 (local address 127.0.0.1:55628)
0806 17:05:30.377531 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:30.377546 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:30.377639 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:30.378309 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:30.378311 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:30.378329 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:30.378414 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:30.378422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.379714 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.379717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:30.380962 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:30.380973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.382097 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.382100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:30.382211 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:30.382239 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 17:05:30.382319 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 17:05:30.382396 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"thread_start_us":106,"threads_started":1}
I20260806 17:05:30.382736 26090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:30.377021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:43757 (local address 127.0.0.1:40886)
0806 17:05:30.377868 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:30.377882 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:30.378020 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:30.378643 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:30.378646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:30.378663 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:30.378745 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:30.378751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.379896 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.379899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:30.381035 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:30.381046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.382314 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.382318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:30.382420 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:30.382445 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 17:05:30.382526 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 17:05:30.382604 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":102,"threads_started":1}
I20260806 17:05:30.382931 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:30.377207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.9.111.67:43757)
0806 17:05:30.377629 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:30.377641 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:30.377959 (+ 318us) server_negotiation.cc:407] Connection header received
0806 17:05:30.378172 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:30.378179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:30.378347 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:30.378459 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:30.378884 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.379726 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.381207 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.382001 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.382101 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:30.382248 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:30.382575 (+ 327us) server_negotiation.cc:299] Negotiation successful
0806 17:05:30.382669 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":128,"threads_started":1}
I20260806 17:05:30.383215 26093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:30.377083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.9.111.65:40329)
0806 17:05:30.377833 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:30.377849 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:30.377873 (+ 24us) server_negotiation.cc:407] Connection header received
0806 17:05:30.377957 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:30.377962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:30.378164 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:30.378312 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:30.378520 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.379544 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.381099 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.381959 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.382810 (+ 851us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:30.382953 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:30.383001 (+ 48us) server_negotiation.cc:299] Negotiation successful
0806 17:05:30.383085 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":153,"threads_started":1}
I20260806 17:05:30.385468 25870 tablet_service.cc:1467] Processing CreateTablet for tablet c7f25047db3245a68b74b47cbcef9d86 (DEFAULT_TABLE table=test-table [id=34bcb54095f64927b8aa97ef7afb231b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:05:30.385804 26009 tablet_service.cc:1467] Processing CreateTablet for tablet c7f25047db3245a68b74b47cbcef9d86 (DEFAULT_TABLE table=test-table [id=34bcb54095f64927b8aa97ef7afb231b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:05:30.385805 25729 tablet_service.cc:1467] Processing CreateTablet for tablet c7f25047db3245a68b74b47cbcef9d86 (DEFAULT_TABLE table=test-table [id=34bcb54095f64927b8aa97ef7afb231b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:05:30.386649 25870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7f25047db3245a68b74b47cbcef9d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:30.386940 25729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7f25047db3245a68b74b47cbcef9d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:30.386950 26009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7f25047db3245a68b74b47cbcef9d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:30.393117 26095 tablet_bootstrap.cc:492] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0: Bootstrap starting.
I20260806 17:05:30.395052 26096 tablet_bootstrap.cc:492] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588: Bootstrap starting.
I20260806 17:05:30.395841 26095 tablet_bootstrap.cc:654] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:30.396574 26095 log.cc:824] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:30.396997 26096 tablet_bootstrap.cc:654] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:30.397241 26097 tablet_bootstrap.cc:492] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700: Bootstrap starting.
I20260806 17:05:30.397926 26096 log.cc:824] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:30.398943 26095 tablet_bootstrap.cc:492] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0: No bootstrap required, opened a new log
I20260806 17:05:30.399152 26095 ts_tablet_manager.cc:1397] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 17:05:30.399660 26097 tablet_bootstrap.cc:654] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:30.399971 26096 tablet_bootstrap.cc:492] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588: No bootstrap required, opened a new log
I20260806 17:05:30.400229 26096 ts_tablet_manager.cc:1397] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 17:05:30.400514 26097 log.cc:824] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:30.402341 26097 tablet_bootstrap.cc:492] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700: No bootstrap required, opened a new log
I20260806 17:05:30.402594 26097 ts_tablet_manager.cc:1397] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 17:05:30.403589 26095 raft_consensus.cc:348] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } }
I20260806 17:05:30.403903 26095 raft_consensus.cc:374] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:30.403975 26095 raft_consensus.cc:729] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cadb4cd97ae44f89701819e2acb73b0, State: Initialized, Role: FOLLOWER
I20260806 17:05:30.404448 26095 consensus_queue.cc:260] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } }
I20260806 17:05:30.404983 26096 raft_consensus.cc:348] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } }
I20260806 17:05:30.405246 26096 raft_consensus.cc:374] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:30.405303 26096 raft_consensus.cc:729] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82a0d759729741da91b829b16df9b588, State: Initialized, Role: FOLLOWER
I20260806 17:05:30.405552 26095 ts_tablet_manager.cc:1428] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 17:05:30.405941 25939 heartbeater.cc:503] Master 127.9.111.126:34923 was elected leader, sending a full tablet report...
I20260806 17:05:30.406142 26097 raft_consensus.cc:348] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } }
I20260806 17:05:30.406376 26097 raft_consensus.cc:374] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:30.406450 26097 raft_consensus.cc:729] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da07027f963c4af49c6472087d25f700, State: Initialized, Role: FOLLOWER
I20260806 17:05:30.405753 26096 consensus_queue.cc:260] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } }
I20260806 17:05:30.406862 26097 consensus_queue.cc:260] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } }
I20260806 17:05:30.407550 26078 heartbeater.cc:503] Master 127.9.111.126:34923 was elected leader, sending a full tablet report...
I20260806 17:05:30.407756 26096 ts_tablet_manager.cc:1428] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 17:05:30.407806 26097 ts_tablet_manager.cc:1428] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 17:05:30.408061 25797 heartbeater.cc:503] Master 127.9.111.126:34923 was elected leader, sending a full tablet report...
I20260806 17:05:30.408339 26104 raft_consensus.cc:482] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:05:30.408782 26104 raft_consensus.cc:504] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } }
I20260806 17:05:30.411296 26104 leader_election.cc:288] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cadb4cd97ae44f89701819e2acb73b0 (127.9.111.66:34373), 82a0d759729741da91b829b16df9b588 (127.9.111.65:40329)
I20260806 17:05:30.414692 26091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:30.410494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:37219 (local address 127.9.111.66:34373)
0806 17:05:30.410728 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:30.410732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:30.410744 (+ 12us) server_negotiation.cc:407] Connection header received
0806 17:05:30.410780 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:30.410783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:30.410830 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:30.410909 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:30.411195 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.411927 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.413495 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.414157 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.414258 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:30.414298 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:30.414536 (+ 238us) server_negotiation.cc:299] Negotiation successful
0806 17:05:30.414603 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260806 17:05:30.414702 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:30.410128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:34373 (local address 127.9.111.67:37219)
0806 17:05:30.410478 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:30.410494 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:30.410577 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:30.410941 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:30.410944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:30.410961 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:30.411070 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:30.411077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.412076 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.412079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:30.413366 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:30.413380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.414267 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.414271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:30.414383 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:30.414409 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 17:05:30.414488 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 17:05:30.414564 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268}
I20260806 17:05:30.415349 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f25047db3245a68b74b47cbcef9d86" candidate_uuid: "da07027f963c4af49c6472087d25f700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cadb4cd97ae44f89701819e2acb73b0" is_pre_election: true
I20260806 17:05:30.415688 25890 raft_consensus.cc:2454] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da07027f963c4af49c6472087d25f700 in term 0.
I20260806 17:05:30.416123 26093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:30.411641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:51485 (local address 127.9.111.65:40329)
0806 17:05:30.411781 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:30.411786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:30.412039 (+ 253us) server_negotiation.cc:407] Connection header received
0806 17:05:30.412132 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:30.412135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:30.412180 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:30.412262 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:30.412502 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.413304 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.414799 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.415398 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.415495 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:30.415541 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:30.415924 (+ 383us) server_negotiation.cc:299] Negotiation successful
0806 17:05:30.416010 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 17:05:30.416179 25964 leader_election.cc:302] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cadb4cd97ae44f89701819e2acb73b0, da07027f963c4af49c6472087d25f700; no voters:
I20260806 17:05:30.416488 26105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:30.411563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:40329 (local address 127.9.111.67:51485)
0806 17:05:30.411946 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:30.411961 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:30.412055 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:30.412291 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:30.412293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:30.412307 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:30.412396 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:30.412400 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.413441 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.413446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:30.414636 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:30.414645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.415654 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.415668 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:30.415776 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:30.415798 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 17:05:30.416272 (+ 474us) client_negotiation.cc:240] Negotiation successful
0806 17:05:30.416350 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":153,"threads_started":1}
I20260806 17:05:30.416486 26104 raft_consensus.cc:2781] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:05:30.416615 26104 raft_consensus.cc:482] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:05:30.416721 26104 raft_consensus.cc:3037] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:30.417217 25750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f25047db3245a68b74b47cbcef9d86" candidate_uuid: "da07027f963c4af49c6472087d25f700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82a0d759729741da91b829b16df9b588" is_pre_election: true
I20260806 17:05:30.417598 25750 raft_consensus.cc:2454] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da07027f963c4af49c6472087d25f700 in term 0.
I20260806 17:05:30.418344 26104 raft_consensus.cc:504] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } }
I20260806 17:05:30.418738 26104 leader_election.cc:288] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [CANDIDATE]: Term 1 election: Requested vote from peers 6cadb4cd97ae44f89701819e2acb73b0 (127.9.111.66:34373), 82a0d759729741da91b829b16df9b588 (127.9.111.65:40329)
I20260806 17:05:30.419145 25750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f25047db3245a68b74b47cbcef9d86" candidate_uuid: "da07027f963c4af49c6472087d25f700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82a0d759729741da91b829b16df9b588"
I20260806 17:05:30.419170 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f25047db3245a68b74b47cbcef9d86" candidate_uuid: "da07027f963c4af49c6472087d25f700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cadb4cd97ae44f89701819e2acb73b0"
I20260806 17:05:30.419297 25750 raft_consensus.cc:3037] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:30.419351 25890 raft_consensus.cc:3037] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:30.420526 25890 raft_consensus.cc:2454] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da07027f963c4af49c6472087d25f700 in term 1.
I20260806 17:05:30.420524 25750 raft_consensus.cc:2454] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da07027f963c4af49c6472087d25f700 in term 1.
I20260806 17:05:30.420909 25962 leader_election.cc:302] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82a0d759729741da91b829b16df9b588, da07027f963c4af49c6472087d25f700; no voters:
I20260806 17:05:30.421111 26104 raft_consensus.cc:2781] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:05:30.421231 26104 raft_consensus.cc:686] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [term 1 LEADER]: Becoming Leader. State: Replica: da07027f963c4af49c6472087d25f700, State: Running, Role: LEADER
I20260806 17:05:30.421614 26104 consensus_queue.cc:237] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } }
I20260806 17:05:30.425573 25597 catalog_manager.cc:5495] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 reported cstate change: term changed from 0 to 1, leader changed from <none> to da07027f963c4af49c6472087d25f700 (127.9.111.67). New cstate: current_term: 1 leader_uuid: "da07027f963c4af49c6472087d25f700" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } health_report { overall_health: HEALTHY } } }
I20260806 17:05:30.447403 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:30.443493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.9.111.67:43757)
0806 17:05:30.443631 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:30.443635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:30.443650 (+ 15us) server_negotiation.cc:407] Connection header received
0806 17:05:30.443801 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:30.443804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:30.443866 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:30.443966 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:05:30.444214 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.444748 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.445640 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:30.445959 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:30.446055 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:30.446423 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:05:30.446897 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:05:30.447001 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:30.447150 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 17:05:30.447212 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 17:05:30.454485 25890 raft_consensus.cc:1261] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0 [term 1 FOLLOWER]: Refusing update from remote peer da07027f963c4af49c6472087d25f700: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:05:30.455308 25750 raft_consensus.cc:1261] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Refusing update from remote peer da07027f963c4af49c6472087d25f700: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:05:30.455399 26103 consensus_queue.cc:1035] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:05:30.456094 26103 consensus_queue.cc:1035] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:05:30.465737 26108 mvcc.cc:205] Tried to move back new op lower bound from 7315603171132862464 to 7315603171010637824. Current Snapshot: MvccSnapshot[applied={T|T < 7315603171132862464}]
I20260806 17:05:30.466073 26110 mvcc.cc:205] Tried to move back new op lower bound from 7315603171132862464 to 7315603171010637824. Current Snapshot: MvccSnapshot[applied={T|T < 7315603171132862464}]
I20260806 17:05:30.468700 26109 mvcc.cc:205] Tried to move back new op lower bound from 7315603171132862464 to 7315603171010637824. Current Snapshot: MvccSnapshot[applied={T|T < 7315603171132862464}]
I20260806 17:05:30.477169 25597 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:05:30.477393 25597 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:05:30.480276 25597 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: Sending DeleteTablet for 3 replicas of tablet c7f25047db3245a68b74b47cbcef9d86
I20260806 17:05:30.481225 26009 tablet_service.cc:1514] Processing DeleteTablet for tablet c7f25047db3245a68b74b47cbcef9d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:05:30 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:40886
I20260806 17:05:30.481242 25729 tablet_service.cc:1514] Processing DeleteTablet for tablet c7f25047db3245a68b74b47cbcef9d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:05:30 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:55628
I20260806 17:05:30.481798 26120 tablet_replica.cc:330] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700: stopping tablet replica
I20260806 17:05:30.481864 25870 tablet_service.cc:1514] Processing DeleteTablet for tablet c7f25047db3245a68b74b47cbcef9d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:05:30 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:35462
I20260806 17:05:30.482043 26121 tablet_replica.cc:330] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588: stopping tablet replica
I20260806 17:05:30.482360 26121 raft_consensus.cc:2229] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:05:30.482352 26120 raft_consensus.cc:2229] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:05:30.482384 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 25662
I20260806 17:05:30.482710 26121 raft_consensus.cc:2258] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:05:30.485066 26120 raft_consensus.cc:2258] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:05:30.487038 26120 ts_tablet_manager.cc:1905] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:05:30.489807 26120 ts_tablet_manager.cc:1918] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:05:30.489965 26120 log.cc:1196] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/c7f25047db3245a68b74b47cbcef9d86
I20260806 17:05:30.490257 26122 tablet_replica.cc:330] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0: stopping tablet replica
I20260806 17:05:30.490343 26120 ts_tablet_manager.cc:1939] T c7f25047db3245a68b74b47cbcef9d86 P da07027f963c4af49c6472087d25f700: Deleting consensus metadata
I20260806 17:05:30.490742 26122 raft_consensus.cc:2229] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:05:30.491123 26122 raft_consensus.cc:2258] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:05:30.491711 25584 catalog_manager.cc:4841] TS da07027f963c4af49c6472087d25f700 (127.9.111.67:43757): tablet c7f25047db3245a68b74b47cbcef9d86 (table test-table [id=34bcb54095f64927b8aa97ef7afb231b]) successfully deleted
I20260806 17:05:30.492622 26122 ts_tablet_manager.cc:1905] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 17:05:30.492990 25582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.111.65:40329 (error 108)
I20260806 17:05:30.493906 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 25804
I20260806 17:05:30.494123 26090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:30.493771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:40329 (local address 127.0.0.1:55632)
0806 17:05:30.493965 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:30.494038 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.65:40329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260806 17:05:30.494282 25582 catalog_manager.cc:4584] TS 82a0d759729741da91b829b16df9b588 (127.9.111.65:40329): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c7f25047db3245a68b74b47cbcef9d86: Network error: Client connection negotiation failed: client connection to 127.9.111.65:40329: connect: Connection refused (error 111)
I20260806 17:05:30.504294 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 25944
I20260806 17:05:30.504537 26090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:30.504168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:34373 (local address 127.0.0.1:35474)
0806 17:05:30.504337 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:30.504412 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.66:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 17:05:30.510596 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 25563
I20260806 17:05:50.518316 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:34923
--webserver_interface=127.9.111.126
--webserver_port=36375
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:34923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:05:50.636035 26124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:50.636402 26124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:50.636488 26124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:50.640764 26124 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 17:05:50.640864 26124 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 17:05:50.640893 26124 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 17:05:50.640918 26124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:05:50.640961 26124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:50.641009 26124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:50.641067 26124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:05:50.641108 26124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:05:50.646726 26124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:34923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:34923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=36375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:50.647917 26124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:50.649019 26124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:50.663309 26131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:50.663384 26133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:50.663304 26130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:50.663458 26124 server_base.cc:1034] running on GCE node
I20260806 17:05:50.664636 26124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:50.665865 26124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:50.667068 26124 hybrid_clock.cc:648] HybridClock initialized: now 1786035950667047 us; error 44 us; skew 500 ppm
Aug 06 17:05:50 dist-test-slave-69rg krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035950, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:50.676671 26124 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:05:50.677855 26124 webserver.cc:460] Webserver started at http://127.9.111.126:36375/ using document root <none> and password file <none>
I20260806 17:05:50.678541 26124 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:50.678670 26124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:50.682862 26124 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 17:05:50.685178 26140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:50.686481 26124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 17:05:50.686640 26124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "d0be2ea560f14f1aa80e061efccb4210"
format_stamp: "Formatted at 2026-08-06 17:05:29 on dist-test-slave-69rg"
server_key: "ea161d1633027cfdef2dac78d80c4be7"
server_key_iv: "a8778cae1d1953c752daa2a268f83fa6"
server_key_version: "encryptionkey@0"
I20260806 17:05:50.687265 26124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:50.706755 26124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:50.710772 26124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:50.711014 26124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:50.718954 26124 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:34923
I20260806 17:05:50.719246 26195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:34923 every 8 connection(s)
I20260806 17:05:50.720155 26124 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:05:50.724748 26196 sys_catalog.cc:263] Verifying existing consensus state
I20260806 17:05:50.726509 26196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: Bootstrap starting.
I20260806 17:05:50.726933 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 26124
I20260806 17:05:50.727388 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:40329
--local_ip_for_outbound_sockets=127.9.111.65
--tserver_master_addrs=127.9.111.126:34923
--webserver_port=41903
--webserver_interface=127.9.111.65
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:05:50.744871 26196 log.cc:824] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:50.754940 26196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 17:05:50.755367 26196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: Bootstrap complete.
I20260806 17:05:50.759691 26196 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0be2ea560f14f1aa80e061efccb4210" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34923 } }
I20260806 17:05:50.760136 26196 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0be2ea560f14f1aa80e061efccb4210, State: Initialized, Role: FOLLOWER
I20260806 17:05:50.760641 26196 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0be2ea560f14f1aa80e061efccb4210" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34923 } }
I20260806 17:05:50.760792 26196 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:05:50.760890 26196 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:05:50.761009 26196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 1 FOLLOWER]: Advancing to term 2
I20260806 17:05:50.763264 26196 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0be2ea560f14f1aa80e061efccb4210" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34923 } }
I20260806 17:05:50.763617 26196 leader_election.cc:302] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0be2ea560f14f1aa80e061efccb4210; no voters:
I20260806 17:05:50.763942 26196 leader_election.cc:288] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 17:05:50.764132 26200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 17:05:50.764405 26200 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 2 LEADER]: Becoming Leader. State: Replica: d0be2ea560f14f1aa80e061efccb4210, State: Running, Role: LEADER
I20260806 17:05:50.764737 26200 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0be2ea560f14f1aa80e061efccb4210" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34923 } }
I20260806 17:05:50.765272 26196 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:05:50.766286 26201 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d0be2ea560f14f1aa80e061efccb4210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0be2ea560f14f1aa80e061efccb4210" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34923 } } }
I20260806 17:05:50.766434 26201 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [sys.catalog]: This master's current role is: LEADER
I20260806 17:05:50.766741 26202 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0be2ea560f14f1aa80e061efccb4210. Latest consensus state: current_term: 2 leader_uuid: "d0be2ea560f14f1aa80e061efccb4210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0be2ea560f14f1aa80e061efccb4210" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34923 } } }
I20260806 17:05:50.766836 26202 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [sys.catalog]: This master's current role is: LEADER
I20260806 17:05:50.767242 26209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:05:50.772351 26209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:05:50.772985 26209 catalog_manager.cc:1259] Loaded cluster ID: 347e20fb9f974d8da40d92dd865491af
I20260806 17:05:50.773051 26209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:05:50.775688 26209 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:05:50.776643 26209 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: Loaded TSK: 0
I20260806 17:05:50.796474 26209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: Generated new TSK 1
I20260806 17:05:50.796799 26209 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 17:05:50.855508 26198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:50.855785 26198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:50.855830 26198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:50.860136 26198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:50.860220 26198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:50.860321 26198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:05:50.866096 26198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:40329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=41903
--tserver_master_addrs=127.9.111.126:34923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:50.867324 26198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:50.868391 26198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:50.878449 26227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:50.878458 26224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:50.878458 26225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:50.879038 26198 server_base.cc:1034] running on GCE node
I20260806 17:05:50.879541 26198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:50.880415 26198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:50.881639 26198 hybrid_clock.cc:648] HybridClock initialized: now 1786035950881611 us; error 37 us; skew 500 ppm
Aug 06 17:05:50 dist-test-slave-69rg krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035950, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:50.885236 26198 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:05:50.886358 26198 webserver.cc:460] Webserver started at http://127.9.111.65:41903/ using document root <none> and password file <none>
I20260806 17:05:50.887336 26198 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:50.887488 26198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:50.892277 26198 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 17:05:50.894835 26234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:50.896219 26198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:05:50.896401 26198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "82a0d759729741da91b829b16df9b588"
format_stamp: "Formatted at 2026-08-06 17:05:29 on dist-test-slave-69rg"
server_key: "2763d4b4003a971e36d45e91173aa8d2"
server_key_iv: "486409e1827fc9090e6bb449e448deee"
server_key_version: "encryptionkey@0"
I20260806 17:05:50.896983 26198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:50.933593 26198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:50.937175 26198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:50.937397 26198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:50.938061 26198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:05:50.939186 26242 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 17:05:50.943589 26198 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 17:05:50.943706 26198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260806 17:05:50.943809 26198 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 17:05:50.945680 26198 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 17:05:50.945756 26198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 17:05:50.946108 26242 tablet_bootstrap.cc:492] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588: Bootstrap starting.
I20260806 17:05:50.956776 26198 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:40329
I20260806 17:05:50.956883 26351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:40329 every 8 connection(s)
I20260806 17:05:50.957844 26198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:05:50.965945 26242 log.cc:824] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:50.966002 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 26198
I20260806 17:05:50.966594 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:34373
--local_ip_for_outbound_sockets=127.9.111.66
--tserver_master_addrs=127.9.111.126:34923
--webserver_port=33437
--webserver_interface=127.9.111.66
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 17:05:50 dist-test-slave-69rg krb5kdc[25549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035950, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:05:50.974917 26242 tablet_bootstrap.cc:492] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 17:05:50.975368 26242 tablet_bootstrap.cc:492] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588: Bootstrap complete.
I20260806 17:05:50.976505 26242 ts_tablet_manager.cc:1397] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.018s
I20260806 17:05:50.976964 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:50.959940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:40049)
0806 17:05:50.960428 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:50.960482 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:50.961309 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:50.962825 (+ 1516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:50.962834 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:50.963294 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:50.963736 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:50.963751 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:50.965611 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:50.965614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:50.967032 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:50.967043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:50.967982 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:50.967988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:50.968174 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:50.968829 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:50.968846 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:50.970492 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:50.973085 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:50.973092 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:50.973106 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:50.973516 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:50.973953 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:50.973957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:50.973961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:50.974175 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:50.974886 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:50.974896 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:50.975218 (+ 322us) client_negotiation.cc:769] Sending connection context
0806 17:05:50.975456 (+ 238us) client_negotiation.cc:240] Negotiation successful
0806 17:05:50.975834 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":151,"threads_started":1}
I20260806 17:05:50.978972 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:50.959977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:40049 (local address 127.9.111.126:34923)
0806 17:05:50.960565 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:50.960591 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:50.960650 (+ 59us) server_negotiation.cc:407] Connection header received
0806 17:05:50.961805 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:50.961846 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:50.962325 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:50.962703 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:50.963933 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:50.965437 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:50.967173 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:50.967833 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:50.968028 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:50.970705 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:50.970711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:50.970726 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:50.970767 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:50.972950 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:50.973651 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:50.973657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:50.973662 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:50.973843 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:50.974295 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:50.974298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:50.974300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:50.974742 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:50.974887 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:50.977631 (+ 2744us) server_negotiation.cc:299] Negotiation successful
0806 17:05:50.978004 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":138,"threads_started":1}
I20260806 17:05:50.978900 26242 raft_consensus.cc:348] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } }
I20260806 17:05:50.979422 26242 raft_consensus.cc:729] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82a0d759729741da91b829b16df9b588, State: Initialized, Role: FOLLOWER
I20260806 17:05:50.979904 26242 consensus_queue.cc:260] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } }
I20260806 17:05:50.980803 26242 ts_tablet_manager.cc:1428] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588: Time spent starting tablet: real 0.004s user 0.005s sys 0.001s
I20260806 17:05:50.981483 26352 heartbeater.cc:347] Connected to a master server at 127.9.111.126:34923
I20260806 17:05:50.981705 26352 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:50.982208 26352 heartbeater.cc:511] Master 127.9.111.126:34923 requested a full tablet report, sending...
I20260806 17:05:50.984628 26152 ts_manager.cc:194] Registered new tserver with Master: 82a0d759729741da91b829b16df9b588 (127.9.111.65:40329)
I20260806 17:05:50.988713 26152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:40049
I20260806 17:05:50.991395 26352 heartbeater.cc:503] Master 127.9.111.126:34923 was elected leader, sending a full tablet report...
Aug 06 17:05:50 dist-test-slave-69rg krb5kdc[25549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035950, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for kudu/127.9.111.65@KRBTEST.COM
I20260806 17:05:50.999214 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:50.988045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:40329 (local address 127.0.0.1:36958)
0806 17:05:50.988764 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:50.988800 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:50.988969 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:50.989575 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:50.989583 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:50.990097 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:50.990231 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:50.990239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:50.991804 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:50.991807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:50.992588 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:50.992598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:50.993239 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:50.993243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:50.993363 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:50.993984 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:50.993995 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:50.995542 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:50.997377 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:50.997387 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:50.997392 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:50.997627 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:50.997989 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:50.997992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:50.997994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:50.998082 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:50.998553 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:50.998559 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:50.998686 (+ 127us) client_negotiation.cc:769] Sending connection context
0806 17:05:50.998827 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 17:05:50.999031 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":122,"threads_started":1}
I20260806 17:05:50.999217 26360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:50.988097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36958 (local address 127.9.111.65:40329)
0806 17:05:50.988532 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:50.988542 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:50.989080 (+ 538us) server_negotiation.cc:407] Connection header received
0806 17:05:50.989141 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:50.989145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:50.989305 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:50.989448 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:50.990372 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:50.991565 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:50.992708 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:50.993116 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:50.993236 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:50.995692 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:50.995701 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:50.995710 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:50.995730 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:50.997262 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:50.997757 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:50.997762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:50.997763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:50.997895 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:50.998190 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:50.998194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:50.998196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:50.998434 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:50.998576 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:50.998909 (+ 333us) server_negotiation.cc:299] Negotiation successful
0806 17:05:50.999074 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":80,"threads_started":1}
I20260806 17:05:51.000397 26285 tablet_service.cc:1514] Processing DeleteTablet for tablet c7f25047db3245a68b74b47cbcef9d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:05:30 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:36958
I20260806 17:05:51.001425 26361 tablet_replica.cc:330] stopping tablet replica
I20260806 17:05:51.001737 26361 raft_consensus.cc:2229] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:05:51.002034 26361 raft_consensus.cc:2258] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:05:51.003046 26361 ts_tablet_manager.cc:1905] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:05:51.006920 26361 ts_tablet_manager.cc:1918] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:05:51.007057 26361 log.cc:1196] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/wal/wals/c7f25047db3245a68b74b47cbcef9d86
I20260806 17:05:51.007349 26361 ts_tablet_manager.cc:1939] T c7f25047db3245a68b74b47cbcef9d86 P 82a0d759729741da91b829b16df9b588: Deleting consensus metadata
I20260806 17:05:51.009341 26143 catalog_manager.cc:4841] TS 82a0d759729741da91b829b16df9b588 (127.9.111.65:40329): tablet c7f25047db3245a68b74b47cbcef9d86 (table test-table [id=34bcb54095f64927b8aa97ef7afb231b]) successfully deleted
W20260806 17:05:51.099637 26357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:51.099931 26357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:51.099977 26357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:51.104189 26357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:51.104270 26357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:51.104333 26357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:05:51.109669 26357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:34373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=33437
--tserver_master_addrs=127.9.111.126:34923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:51.110705 26357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:51.111734 26357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:51.121441 26369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:51.121425 26366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:51.121598 26357 server_base.cc:1034] running on GCE node
W20260806 17:05:51.121798 26367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:51.122225 26357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:51.122824 26357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:51.123988 26357 hybrid_clock.cc:648] HybridClock initialized: now 1786035951123968 us; error 32 us; skew 500 ppm
Aug 06 17:05:51 dist-test-slave-69rg krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035951, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:51.126991 26357 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:05:51.127930 26357 webserver.cc:460] Webserver started at http://127.9.111.66:33437/ using document root <none> and password file <none>
I20260806 17:05:51.128460 26357 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:51.128537 26357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:51.132462 26357 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 17:05:51.134665 26376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:51.135942 26357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:05:51.136076 26357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "6cadb4cd97ae44f89701819e2acb73b0"
format_stamp: "Formatted at 2026-08-06 17:05:30 on dist-test-slave-69rg"
server_key: "052df19c75270527c8d4758305f62030"
server_key_iv: "642cdf39d5eacfc67dacbd5d9f503421"
server_key_version: "encryptionkey@0"
I20260806 17:05:51.136615 26357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:51.151558 26357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:51.155208 26357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:51.155429 26357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:51.156122 26357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:05:51.157202 26383 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 17:05:51.161531 26357 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 17:05:51.161656 26357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260806 17:05:51.161749 26357 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 17:05:51.163604 26357 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 17:05:51.163684 26357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 17:05:51.163954 26383 tablet_bootstrap.cc:492] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0: Bootstrap starting.
I20260806 17:05:51.169493 26383 log.cc:824] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:51.175034 26357 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:34373
I20260806 17:05:51.175112 26495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:34373 every 8 connection(s)
I20260806 17:05:51.175936 26357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:05:51.176203 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 26357
I20260806 17:05:51.176738 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:43757
--local_ip_for_outbound_sockets=127.9.111.67
--tserver_master_addrs=127.9.111.126:34923
--webserver_port=36881
--webserver_interface=127.9.111.67
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:05:51.178478 26383 tablet_bootstrap.cc:492] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 17:05:51.178956 26383 tablet_bootstrap.cc:492] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0: Bootstrap complete.
I20260806 17:05:51.179363 26383 ts_tablet_manager.cc:1397] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260806 17:05:51.183385 26383 raft_consensus.cc:348] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } }
I20260806 17:05:51.184052 26383 raft_consensus.cc:729] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cadb4cd97ae44f89701819e2acb73b0, State: Initialized, Role: FOLLOWER
I20260806 17:05:51.184664 26383 consensus_queue.cc:260] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } }
I20260806 17:05:51.185899 26383 ts_tablet_manager.cc:1428] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
Aug 06 17:05:51 dist-test-slave-69rg krb5kdc[25549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035951, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:05:51.191357 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:51.178275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:48869 (local address 127.9.111.126:34923)
0806 17:05:51.178419 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:51.178423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:51.179657 (+ 1234us) server_negotiation.cc:407] Connection header received
0806 17:05:51.180629 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:51.180632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:51.180700 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:51.180812 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:51.181982 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.182631 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.183837 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.184146 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.184278 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:51.187434 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:51.187445 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:51.187449 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:51.187477 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:51.189070 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:51.189667 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:51.189671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:51.189672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:51.189757 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:51.190171 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:51.190175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:51.190176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:51.190385 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:51.190500 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:51.191029 (+ 529us) server_negotiation.cc:299] Negotiation successful
0806 17:05:51.191184 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 17:05:51.192239 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:51.178425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:48869)
0806 17:05:51.179501 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:51.179552 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:51.180419 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:51.180967 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:51.180974 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:51.181320 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:51.181781 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:51.181794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.182829 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.182831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:51.183697 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:51.183706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.184270 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.184274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:51.184460 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:51.185568 (+ 1108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:51.185588 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:51.187220 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:51.189211 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:51.189218 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:51.189235 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:51.189534 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:51.189900 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:51.189904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:51.189906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:51.190065 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:51.190498 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:51.190505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:51.190776 (+ 271us) client_negotiation.cc:769] Sending connection context
0806 17:05:51.190986 (+ 210us) client_negotiation.cc:240] Negotiation successful
0806 17:05:51.191372 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"thread_start_us":172,"threads_started":1}
I20260806 17:05:51.193503 26496 heartbeater.cc:347] Connected to a master server at 127.9.111.126:34923
I20260806 17:05:51.193763 26496 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:51.194324 26496 heartbeater.cc:511] Master 127.9.111.126:34923 requested a full tablet report, sending...
I20260806 17:05:51.196113 26152 ts_manager.cc:194] Registered new tserver with Master: 6cadb4cd97ae44f89701819e2acb73b0 (127.9.111.66:34373)
I20260806 17:05:51.197499 26152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:48869
I20260806 17:05:51.200065 26496 heartbeater.cc:503] Master 127.9.111.126:34923 was elected leader, sending a full tablet report...
Aug 06 17:05:51 dist-test-slave-69rg krb5kdc[25549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035950, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for kudu/127.9.111.66@KRBTEST.COM
I20260806 17:05:51.208174 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:51.197247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:34373 (local address 127.0.0.1:60048)
0806 17:05:51.197382 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:51.197400 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:51.197492 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:51.198367 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:51.198370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:51.198582 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:51.198695 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:51.198701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.200071 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.200077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:51.200944 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:51.200957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.201878 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.201882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:51.201991 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:51.202576 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:51.202595 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:51.204045 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:51.206221 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:51.206225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:51.206227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:51.206515 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:51.206881 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:51.206884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:51.206886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:51.206986 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:51.207598 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:51.207601 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:51.207715 (+ 114us) client_negotiation.cc:769] Sending connection context
0806 17:05:51.207830 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 17:05:51.207995 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 17:05:51.208302 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:51.197492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.9.111.66:34373)
0806 17:05:51.197892 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:51.197905 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:51.197931 (+ 26us) server_negotiation.cc:407] Connection header received
0806 17:05:51.197996 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:51.198001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:51.198213 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:51.198357 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:51.198843 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.199916 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.201079 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.201716 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.201860 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:51.204258 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:51.204268 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:51.204279 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:51.204303 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:51.206097 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:51.206658 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:51.206664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:51.206665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:51.206763 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:51.207115 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:51.207118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:51.207119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:51.207479 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:51.207608 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:51.207946 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 17:05:51.208097 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":173,"threads_started":1}
I20260806 17:05:51.209347 26427 tablet_service.cc:1514] Processing DeleteTablet for tablet c7f25047db3245a68b74b47cbcef9d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:05:30 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:60048
I20260806 17:05:51.210438 26502 tablet_replica.cc:330] stopping tablet replica
I20260806 17:05:51.210747 26502 raft_consensus.cc:2229] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:05:51.211002 26502 raft_consensus.cc:2258] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:05:51.212042 26502 ts_tablet_manager.cc:1905] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:05:51.216121 26502 ts_tablet_manager.cc:1918] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:05:51.216223 26502 log.cc:1196] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/c7f25047db3245a68b74b47cbcef9d86
I20260806 17:05:51.216506 26502 ts_tablet_manager.cc:1939] T c7f25047db3245a68b74b47cbcef9d86 P 6cadb4cd97ae44f89701819e2acb73b0: Deleting consensus metadata
I20260806 17:05:51.218482 26145 catalog_manager.cc:4841] TS 6cadb4cd97ae44f89701819e2acb73b0 (127.9.111.66:34373): tablet c7f25047db3245a68b74b47cbcef9d86 (table test-table [id=34bcb54095f64927b8aa97ef7afb231b]) successfully deleted
W20260806 17:05:51.302453 26498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:05:51.302742 26498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:05:51.302795 26498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:05:51.306934 26498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:05:51.307000 26498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:05:51.307061 26498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:05:51.312485 26498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:43757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=36881
--tserver_master_addrs=127.9.111.126:34923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:05:51.313548 26498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:05:51.314601 26498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:05:51.324388 26510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:51.324388 26508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:05:51.324410 26507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:05:51.324453 26498 server_base.cc:1034] running on GCE node
I20260806 17:05:51.325069 26498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:05:51.325742 26498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:05:51.326967 26498 hybrid_clock.cc:648] HybridClock initialized: now 1786035951326930 us; error 47 us; skew 500 ppm
Aug 06 17:05:51 dist-test-slave-69rg krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035951, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:05:51.330195 26498 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:05:51.331226 26498 webserver.cc:460] Webserver started at http://127.9.111.67:36881/ using document root <none> and password file <none>
I20260806 17:05:51.331820 26498 fs_manager.cc:359] Metadata directory not provided
I20260806 17:05:51.331938 26498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:05:51.336002 26498 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 17:05:51.338332 26517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:51.339676 26498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:05:51.339866 26498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "da07027f963c4af49c6472087d25f700"
format_stamp: "Formatted at 2026-08-06 17:05:30 on dist-test-slave-69rg"
server_key: "9768aee8ef24d9f869aaae47f0f6ab31"
server_key_iv: "2d66d1c2e23f1362c8481c3c1834c1ed"
server_key_version: "encryptionkey@0"
I20260806 17:05:51.340442 26498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:05:51.356402 26498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:05:51.359745 26498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:05:51.359966 26498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:05:51.360596 26498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:05:51.361465 26498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:05:51.361539 26498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:51.361611 26498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:05:51.361653 26498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:05:51.371989 26498 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:43757
I20260806 17:05:51.372066 26633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:43757 every 8 connection(s)
I20260806 17:05:51.373143 26498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:05:51.375972 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 26498
Aug 06 17:05:51 dist-test-slave-69rg krb5kdc[25549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035951, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:05:51.385654 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:51.376840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44594 (local address 127.9.111.126:34923)
0806 17:05:51.377176 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:51.377180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:51.377285 (+ 105us) server_negotiation.cc:407] Connection header received
0806 17:05:51.377391 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:51.377393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:51.377439 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:51.377519 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:51.378570 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.379133 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.380107 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.380353 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.380431 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:51.382076 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:51.382079 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:51.382081 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:51.382101 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:51.383685 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:51.384189 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:51.384192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:51.384193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:51.384281 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:51.384582 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:51.384585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:51.384586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:51.384853 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:51.385055 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:51.385311 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 17:05:51.385450 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":75,"threads_started":1}
I20260806 17:05:51.388422 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:51.375206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:37889 (local address 127.9.111.126:34923)
0806 17:05:51.375397 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:51.375402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:51.376627 (+ 1225us) server_negotiation.cc:407] Connection header received
0806 17:05:51.377890 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:51.377892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:51.377925 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:51.377994 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:51.379379 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.379878 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.381391 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.381592 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.381669 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:51.384559 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:51.384564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:51.384568 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:51.384592 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:51.386279 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:51.386900 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:51.386903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:51.386905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:51.386996 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:51.387332 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:51.387335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:51.387337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:51.387550 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:51.387651 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:51.388198 (+ 547us) server_negotiation.cc:299] Negotiation successful
0806 17:05:51.388299 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 17:05:51.389384 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:51.375634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.67:37889)
0806 17:05:51.376437 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:51.376494 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:51.377600 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:51.378220 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:51.378230 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:51.378705 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:05:51.379223 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:51.379244 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.380007 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.380011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:51.381263 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:51.381275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.381756 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.381761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:51.382006 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:51.382612 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:05:51.382637 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:05:51.384380 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:05:51.386391 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:51.386396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:51.386411 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:51.386780 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:51.387086 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:05:51.387089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:05:51.387090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:05:51.387226 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:05:51.387671 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:05:51.387676 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:05:51.387973 (+ 297us) client_negotiation.cc:769] Sending connection context
0806 17:05:51.388133 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 17:05:51.388420 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":174,"threads_started":1}
I20260806 17:05:51.390704 26634 heartbeater.cc:347] Connected to a master server at 127.9.111.126:34923
I20260806 17:05:51.390944 26634 heartbeater.cc:464] Registering TS with master...
I20260806 17:05:51.391412 26634 heartbeater.cc:511] Master 127.9.111.126:34923 requested a full tablet report, sending...
I20260806 17:05:51.392460 26152 ts_manager.cc:194] Registered new tserver with Master: da07027f963c4af49c6472087d25f700 (127.9.111.67:43757)
I20260806 17:05:51.393038 26152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:37889
I20260806 17:05:51.399690 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 17:05:51.404312 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:51.400455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44598 (local address 127.9.111.126:34923)
0806 17:05:51.400614 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:51.400620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:51.400812 (+ 192us) server_negotiation.cc:407] Connection header received
0806 17:05:51.401039 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:51.401043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:51.401175 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:51.401299 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:05:51.401560 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.402177 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.403069 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.403293 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.403400 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:51.403734 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:05:51.403878 (+ 144us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 17:05:51.404167 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44598: token expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 17:05:51.404376 26354 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44598: token expired
W20260806 17:05:51.404731 26084 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.111.126:34923)
I20260806 17:05:51.412192 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:51.405111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.9.111.126:34923)
0806 17:05:51.405221 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:51.405226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:51.405300 (+ 74us) server_negotiation.cc:407] Connection header received
0806 17:05:51.405429 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:51.405431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:51.405475 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:51.405557 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:05:51.406272 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.406748 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.407541 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.407759 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.407863 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:51.409216 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:05:51.409219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:05:51.409221 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:05:51.409244 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:05:51.410531 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:51.410969 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:51.410978 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:51.410979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:51.411055 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:05:51.411343 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:05:51.411347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:05:51.411349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:05:51.411529 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:05:51.411671 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:51.411907 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 17:05:51.412023 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 17:05:51.414829 26152 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:05:51.418457 26152 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:05:51.426419 26285 tablet_service.cc:1467] Processing CreateTablet for tablet 77ca99084038479db0d7d3677562d6d8 (DEFAULT_TABLE table=test-table [id=3bb98bbb51c246ccaa32de6650d894d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:05:51.426405 26427 tablet_service.cc:1467] Processing CreateTablet for tablet 77ca99084038479db0d7d3677562d6d8 (DEFAULT_TABLE table=test-table [id=3bb98bbb51c246ccaa32de6650d894d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:05:51.427063 26427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ca99084038479db0d7d3677562d6d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:51.427063 26285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ca99084038479db0d7d3677562d6d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:51.429905 26383 tablet_bootstrap.cc:492] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0: Bootstrap starting.
I20260806 17:05:51.430146 26242 tablet_bootstrap.cc:492] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588: Bootstrap starting.
I20260806 17:05:51.430991 26383 tablet_bootstrap.cc:654] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:51.431061 26242 tablet_bootstrap.cc:654] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:51.431672 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:51.425537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:43757 (local address 127.0.0.1:34856)
0806 17:05:51.425707 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:51.425728 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:51.425808 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:51.427061 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:51.427064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:51.427087 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:51.427203 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:51.427210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.428601 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.428612 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:51.429918 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:51.429929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.431144 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.431151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:51.431262 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:51.431327 (+ 65us) client_negotiation.cc:769] Sending connection context
0806 17:05:51.431394 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 17:05:51.431501 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 17:05:51.432360 26643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:51.426119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34856 (local address 127.9.111.67:43757)
0806 17:05:51.426520 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:51.426533 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:51.426560 (+ 27us) server_negotiation.cc:407] Connection header received
0806 17:05:51.426642 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:51.426648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:51.426881 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:51.427051 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:51.427344 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.428472 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.430102 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.430993 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.431867 (+ 874us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:51.432037 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:51.432094 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 17:05:51.432198 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":169,"threads_started":1}
I20260806 17:05:51.432739 26383 tablet_bootstrap.cc:492] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0: No bootstrap required, opened a new log
I20260806 17:05:51.432832 26242 tablet_bootstrap.cc:492] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588: No bootstrap required, opened a new log
I20260806 17:05:51.432840 26383 ts_tablet_manager.cc:1397] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 17:05:51.432904 26242 ts_tablet_manager.cc:1397] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260806 17:05:51.433318 26383 raft_consensus.cc:348] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } }
I20260806 17:05:51.433318 26242 raft_consensus.cc:348] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } }
I20260806 17:05:51.433445 26242 raft_consensus.cc:374] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:51.433445 26383 raft_consensus.cc:374] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:51.433490 26242 raft_consensus.cc:729] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82a0d759729741da91b829b16df9b588, State: Initialized, Role: FOLLOWER
I20260806 17:05:51.433490 26383 raft_consensus.cc:729] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cadb4cd97ae44f89701819e2acb73b0, State: Initialized, Role: FOLLOWER
I20260806 17:05:51.433633 26242 consensus_queue.cc:260] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } }
I20260806 17:05:51.433656 26383 consensus_queue.cc:260] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } }
I20260806 17:05:51.433946 26383 ts_tablet_manager.cc:1428] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 17:05:51.433946 26242 ts_tablet_manager.cc:1428] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260806 17:05:51.437541 26567 tablet_service.cc:1467] Processing CreateTablet for tablet 77ca99084038479db0d7d3677562d6d8 (DEFAULT_TABLE table=test-table [id=3bb98bbb51c246ccaa32de6650d894d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:05:51.438853 26567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ca99084038479db0d7d3677562d6d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:05:51.445968 26646 tablet_bootstrap.cc:492] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700: Bootstrap starting.
I20260806 17:05:51.448364 26646 tablet_bootstrap.cc:654] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700: Neither blocks nor log segments found. Creating new log.
I20260806 17:05:51.449115 26646 log.cc:824] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700: Log is configured to *not* fsync() on all Append() calls
I20260806 17:05:51.450959 26646 tablet_bootstrap.cc:492] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700: No bootstrap required, opened a new log
I20260806 17:05:51.451123 26646 ts_tablet_manager.cc:1397] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 17:05:51.453972 26646 raft_consensus.cc:348] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } }
I20260806 17:05:51.454213 26646 raft_consensus.cc:374] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:05:51.454260 26646 raft_consensus.cc:729] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da07027f963c4af49c6472087d25f700, State: Initialized, Role: FOLLOWER
I20260806 17:05:51.454623 26646 consensus_queue.cc:260] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } }
I20260806 17:05:51.455251 26634 heartbeater.cc:503] Master 127.9.111.126:34923 was elected leader, sending a full tablet report...
I20260806 17:05:51.455506 26646 ts_tablet_manager.cc:1428] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260806 17:05:51.459132 26353 tablet.cc:2369] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:05:51.502625 26648 raft_consensus.cc:482] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:05:51.502892 26648 raft_consensus.cc:504] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } }
I20260806 17:05:51.504266 26648 leader_election.cc:288] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82a0d759729741da91b829b16df9b588 (127.9.111.65:40329), 6cadb4cd97ae44f89701819e2acb73b0 (127.9.111.66:34373)
I20260806 17:05:51.508701 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:51.504627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:39887 (local address 127.9.111.65:40329)
0806 17:05:51.505026 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:51.505031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:51.505261 (+ 230us) server_negotiation.cc:407] Connection header received
0806 17:05:51.505415 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:51.505418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:51.505483 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:51.505594 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:51.505884 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.506395 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.507450 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.507919 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.508059 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:51.508241 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:51.508437 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 17:05:51.508519 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":121,"threads_started":1}
I20260806 17:05:51.508709 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:51.504651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:48701 (local address 127.9.111.66:34373)
0806 17:05:51.504924 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:51.504929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:51.504946 (+ 17us) server_negotiation.cc:407] Connection header received
0806 17:05:51.505006 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:51.505010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:51.505089 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:51.505230 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:05:51.505571 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.506246 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.507407 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.507912 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.508062 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:51.508242 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:51.508431 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 17:05:51.508517 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260806 17:05:51.508728 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:51.504441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:40329 (local address 127.9.111.67:39887)
0806 17:05:51.505164 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:51.505175 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:51.505298 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:51.505620 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:51.505623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:51.505638 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:51.505753 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:51.505759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.506512 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.506517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:51.507317 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:51.507326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.508052 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.508056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:51.508234 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:51.508326 (+ 92us) client_negotiation.cc:769] Sending connection context
0806 17:05:51.508419 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 17:05:51.508519 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":427,"thread_start_us":131,"threads_started":1}
I20260806 17:05:51.508738 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:51.504441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:34373 (local address 127.9.111.67:48701)
0806 17:05:51.504675 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:51.504692 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:05:51.504791 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:05:51.505246 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:05:51.505249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:05:51.505270 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:05:51.505432 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:51.505438 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.506375 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.506382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:51.507251 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:05:51.507261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.508054 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.508058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:05:51.508220 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:51.508312 (+ 92us) client_negotiation.cc:769] Sending connection context
0806 17:05:51.508413 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 17:05:51.508519 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":36,"spinlock_wait_cycles":41216}
I20260806 17:05:51.509443 26448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ca99084038479db0d7d3677562d6d8" candidate_uuid: "da07027f963c4af49c6472087d25f700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cadb4cd97ae44f89701819e2acb73b0" is_pre_election: true
I20260806 17:05:51.509528 26305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ca99084038479db0d7d3677562d6d8" candidate_uuid: "da07027f963c4af49c6472087d25f700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82a0d759729741da91b829b16df9b588" is_pre_election: true
I20260806 17:05:51.509856 26305 raft_consensus.cc:2454] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da07027f963c4af49c6472087d25f700 in term 0.
I20260806 17:05:51.509856 26448 raft_consensus.cc:2454] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da07027f963c4af49c6472087d25f700 in term 0.
I20260806 17:05:51.510486 26521 leader_election.cc:302] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6cadb4cd97ae44f89701819e2acb73b0, 82a0d759729741da91b829b16df9b588, da07027f963c4af49c6472087d25f700; no voters:
I20260806 17:05:51.510814 26648 raft_consensus.cc:2781] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:05:51.510905 26648 raft_consensus.cc:482] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:05:51.510949 26648 raft_consensus.cc:3037] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:51.512017 26648 raft_consensus.cc:504] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } }
I20260806 17:05:51.512384 26648 leader_election.cc:288] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [CANDIDATE]: Term 1 election: Requested vote from peers 82a0d759729741da91b829b16df9b588 (127.9.111.65:40329), 6cadb4cd97ae44f89701819e2acb73b0 (127.9.111.66:34373)
I20260806 17:05:51.512816 26305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ca99084038479db0d7d3677562d6d8" candidate_uuid: "da07027f963c4af49c6472087d25f700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82a0d759729741da91b829b16df9b588"
I20260806 17:05:51.512799 26448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ca99084038479db0d7d3677562d6d8" candidate_uuid: "da07027f963c4af49c6472087d25f700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cadb4cd97ae44f89701819e2acb73b0"
I20260806 17:05:51.512948 26305 raft_consensus.cc:3037] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:51.512948 26448 raft_consensus.cc:3037] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:05:51.514297 26305 raft_consensus.cc:2454] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da07027f963c4af49c6472087d25f700 in term 1.
I20260806 17:05:51.514300 26448 raft_consensus.cc:2454] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da07027f963c4af49c6472087d25f700 in term 1.
I20260806 17:05:51.514678 26519 leader_election.cc:302] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82a0d759729741da91b829b16df9b588, da07027f963c4af49c6472087d25f700; no voters:
I20260806 17:05:51.514935 26648 raft_consensus.cc:2781] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:05:51.515168 26648 raft_consensus.cc:686] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [term 1 LEADER]: Becoming Leader. State: Replica: da07027f963c4af49c6472087d25f700, State: Running, Role: LEADER
I20260806 17:05:51.515614 26648 consensus_queue.cc:237] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } }
I20260806 17:05:51.519169 26158 catalog_manager.cc:5495] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 reported cstate change: term changed from 0 to 1, leader changed from <none> to da07027f963c4af49c6472087d25f700 (127.9.111.67). New cstate: current_term: 1 leader_uuid: "da07027f963c4af49c6472087d25f700" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } health_report { overall_health: UNKNOWN } } }
I20260806 17:05:51.532387 26643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:51.528793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.9.111.67:43757)
0806 17:05:51.528910 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 17:05:51.528913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:05:51.528927 (+ 14us) server_negotiation.cc:407] Connection header received
0806 17:05:51.529039 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:05:51.529041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:05:51.529095 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:05:51.529194 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:05:51.529432 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.530066 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.530908 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:05:51.531147 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:05:51.531237 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:05:51.531548 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:05:51.532008 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:05:51.532181 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 17:05:51.532211 (+ 30us) server_negotiation.cc:299] Negotiation successful
0806 17:05:51.532266 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 17:05:51.538655 26448 raft_consensus.cc:1261] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0 [term 1 FOLLOWER]: Refusing update from remote peer da07027f963c4af49c6472087d25f700: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:05:51.538655 26305 raft_consensus.cc:1261] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Refusing update from remote peer da07027f963c4af49c6472087d25f700: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:05:51.539361 26651 consensus_queue.cc:1035] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:05:51.539541 26648 consensus_queue.cc:1035] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:05:51.556602 26158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:05:51.556816 26158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:05:51.559280 26158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: Sending DeleteTablet for 3 replicas of tablet 77ca99084038479db0d7d3677562d6d8
I20260806 17:05:51.560011 26285 tablet_service.cc:1514] Processing DeleteTablet for tablet 77ca99084038479db0d7d3677562d6d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:05:51 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:36958
I20260806 17:05:51.560155 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 26198
I20260806 17:05:51.560233 26427 tablet_service.cc:1514] Processing DeleteTablet for tablet 77ca99084038479db0d7d3677562d6d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:05:51 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:60048
I20260806 17:05:51.560518 26567 tablet_service.cc:1514] Processing DeleteTablet for tablet 77ca99084038479db0d7d3677562d6d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:05:51 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:34856
I20260806 17:05:51.561393 26502 tablet_replica.cc:330] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0: stopping tablet replica
I20260806 17:05:51.561518 26502 raft_consensus.cc:2229] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:05:51.561653 26502 raft_consensus.cc:2258] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:05:51.562810 26502 ts_tablet_manager.cc:1905] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:05:51.565073 26502 ts_tablet_manager.cc:1918] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:05:51.565167 26502 log.cc:1196] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/77ca99084038479db0d7d3677562d6d8
I20260806 17:05:51.565418 26502 ts_tablet_manager.cc:1939] T 77ca99084038479db0d7d3677562d6d8 P 6cadb4cd97ae44f89701819e2acb73b0: Deleting consensus metadata
I20260806 17:05:51.566089 26664 tablet_replica.cc:330] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700: stopping tablet replica
I20260806 17:05:51.566531 26664 raft_consensus.cc:2229] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:05:51.566538 26145 catalog_manager.cc:4841] TS 6cadb4cd97ae44f89701819e2acb73b0 (127.9.111.66:34373): tablet 77ca99084038479db0d7d3677562d6d8 (table test-table [id=3bb98bbb51c246ccaa32de6650d894d3]) successfully deleted
I20260806 17:05:51.567143 26664 raft_consensus.cc:2258] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:05:51.568449 26664 ts_tablet_manager.cc:1905] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 17:05:51.569393 26143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.111.65:40329 (error 108)
I20260806 17:05:51.570003 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 26357
I20260806 17:05:51.572088 26664 ts_tablet_manager.cc:1918] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:05:51.572192 26664 log.cc:1196] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/77ca99084038479db0d7d3677562d6d8
I20260806 17:05:51.572552 26664 ts_tablet_manager.cc:1939] T 77ca99084038479db0d7d3677562d6d8 P da07027f963c4af49c6472087d25f700: Deleting consensus metadata
I20260806 17:05:51.572639 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:05:51.572234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:40329 (local address 127.0.0.1:36964)
0806 17:05:51.572436 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 17:05:51.572532 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.65:40329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260806 17:05:51.573120 26143 catalog_manager.cc:4584] TS 82a0d759729741da91b829b16df9b588 (127.9.111.65:40329): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 77ca99084038479db0d7d3677562d6d8: Network error: Client connection negotiation failed: client connection to 127.9.111.65:40329: connect: Connection refused (error 111)
I20260806 17:05:51.573899 26145 catalog_manager.cc:4841] TS da07027f963c4af49c6472087d25f700 (127.9.111.67:43757): tablet 77ca99084038479db0d7d3677562d6d8 (table test-table [id=3bb98bbb51c246ccaa32de6650d894d3]) successfully deleted
I20260806 17:05:51.577551 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 26498
I20260806 17:05:51.583508 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 26124
I20260806 17:06:11.591378 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:34923
--webserver_interface=127.9.111.126
--webserver_port=36375
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:34923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:11.743990 26666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:11.744556 26666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:11.744912 26666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:11.750834 26666 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 17:06:11.750988 26666 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 17:06:11.751021 26666 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 17:06:11.751045 26666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:11.751073 26666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:11.751112 26666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:11.751151 26666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:11.751169 26666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:11.757987 26666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:34923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:34923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=36375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:11.760582 26666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:11.762877 26666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:11.779042 26674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:11.779325 26672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:11.779325 26671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:11.779129 26666 server_base.cc:1034] running on GCE node
I20260806 17:06:11.780812 26666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:11.782655 26666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:11.786219 26666 hybrid_clock.cc:648] HybridClock initialized: now 1786035971786126 us; error 97 us; skew 500 ppm
Aug 06 17:06:11 dist-test-slave-69rg krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035971, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:11.791020 26666 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:06:11.792199 26666 webserver.cc:460] Webserver started at http://127.9.111.126:36375/ using document root <none> and password file <none>
I20260806 17:06:11.792825 26666 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:11.792897 26666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:11.797668 26666 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 17:06:11.800909 26681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:11.803188 26666 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 17:06:11.803483 26666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "d0be2ea560f14f1aa80e061efccb4210"
format_stamp: "Formatted at 2026-08-06 17:05:29 on dist-test-slave-69rg"
server_key: "ea161d1633027cfdef2dac78d80c4be7"
server_key_iv: "a8778cae1d1953c752daa2a268f83fa6"
server_key_version: "encryptionkey@0"
I20260806 17:06:11.804163 26666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:11.823144 26666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:11.826885 26666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:11.827102 26666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:11.835733 26666 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:34923
I20260806 17:06:11.836174 26734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:34923 every 8 connection(s)
I20260806 17:06:11.837044 26666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:11.842940 26735 sys_catalog.cc:263] Verifying existing consensus state
I20260806 17:06:11.843261 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 26666
I20260806 17:06:11.843775 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:40329
--local_ip_for_outbound_sockets=127.9.111.65
--tserver_master_addrs=127.9.111.126:34923
--webserver_port=41903
--webserver_interface=127.9.111.65
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:06:11.845927 26735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: Bootstrap starting.
I20260806 17:06:11.880788 26735 log.cc:824] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:11.901645 26735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 17:06:11.902673 26735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: Bootstrap complete.
I20260806 17:06:11.911880 26735 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0be2ea560f14f1aa80e061efccb4210" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34923 } }
I20260806 17:06:11.912827 26735 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0be2ea560f14f1aa80e061efccb4210, State: Initialized, Role: FOLLOWER
I20260806 17:06:11.913527 26735 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0be2ea560f14f1aa80e061efccb4210" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34923 } }
I20260806 17:06:11.913790 26735 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:11.913976 26735 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:11.914088 26735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 2 FOLLOWER]: Advancing to term 3
I20260806 17:06:11.918821 26735 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0be2ea560f14f1aa80e061efccb4210" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34923 } }
I20260806 17:06:11.919556 26735 leader_election.cc:302] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0be2ea560f14f1aa80e061efccb4210; no voters:
I20260806 17:06:11.920090 26735 leader_election.cc:288] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 17:06:11.920308 26739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 17:06:11.920646 26739 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [term 3 LEADER]: Becoming Leader. State: Replica: d0be2ea560f14f1aa80e061efccb4210, State: Running, Role: LEADER
I20260806 17:06:11.921058 26739 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0be2ea560f14f1aa80e061efccb4210" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34923 } }
I20260806 17:06:11.922269 26735 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:11.922814 26740 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d0be2ea560f14f1aa80e061efccb4210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0be2ea560f14f1aa80e061efccb4210" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34923 } } }
I20260806 17:06:11.922753 26741 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0be2ea560f14f1aa80e061efccb4210. Latest consensus state: current_term: 3 leader_uuid: "d0be2ea560f14f1aa80e061efccb4210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0be2ea560f14f1aa80e061efccb4210" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34923 } } }
I20260806 17:06:11.922937 26740 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:11.922936 26741 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:11.923547 26743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:11.930047 26743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:11.931429 26743 catalog_manager.cc:1259] Loaded cluster ID: 347e20fb9f974d8da40d92dd865491af
I20260806 17:06:11.931603 26743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:11.934706 26743 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:11.935909 26743 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: Loaded TSK: 0
I20260806 17:06:11.936131 26743 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d0be2ea560f14f1aa80e061efccb4210: Loaded TSK: 1
F20260806 17:06:11.937125 26743 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f73c13829ad google::LogMessage::Fail() at ??:0
@ 0x7f73c138588a google::LogMessage::SendToLog() at ??:0
@ 0x7f73c13824de google::LogMessage::Flush() at ??:0
@ 0x7f73c1383a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f73c404618d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f73c403ef0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f73cbce77bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f73cbcc199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f73cbcc2178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f73cbd03df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f73cbd2282b std::function<>::operator()() at ??:0
@ 0x7f73cbcc1b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f73cbcc2f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f73cbcbf579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f73cbd03640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f73c35d43d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f73c35d4c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f73c35d63f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f73c35c5512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f73c4c096db start_thread at ??:0
@ 0x7f73bfe8061f clone at ??:0
W20260806 17:06:12.065930 26737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:12.066366 26737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:12.066444 26737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:12.075208 26737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:12.075295 26737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:12.075387 26737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:06:12.085798 26737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:40329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=41903
--tserver_master_addrs=127.9.111.126:34923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:12.087958 26737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:12.090539 26737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:12.106228 26787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:12.106276 26737 server_base.cc:1034] running on GCE node
W20260806 17:06:12.106223 26784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:12.106223 26785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:12.107074 26737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:12.107846 26737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:12.109381 26737 hybrid_clock.cc:648] HybridClock initialized: now 1786035972109383 us; error 109 us; skew 500 ppm
Aug 06 17:06:12 dist-test-slave-69rg krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035972, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:12.113945 26737 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:06:12.115096 26737 webserver.cc:460] Webserver started at http://127.9.111.65:41903/ using document root <none> and password file <none>
I20260806 17:06:12.115769 26737 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:12.115859 26737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:12.122331 26737 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 17:06:12.125339 26794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:12.127223 26737 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 17:06:12.127425 26737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "82a0d759729741da91b829b16df9b588"
format_stamp: "Formatted at 2026-08-06 17:05:29 on dist-test-slave-69rg"
server_key: "2763d4b4003a971e36d45e91173aa8d2"
server_key_iv: "486409e1827fc9090e6bb449e448deee"
server_key_version: "encryptionkey@0"
I20260806 17:06:12.128067 26737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:12.164057 26737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:12.168475 26737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:12.168994 26737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:12.169942 26737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:12.172463 26801 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 17:06:12.180959 26737 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 17:06:12.181119 26737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20260806 17:06:12.181342 26737 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 17:06:12.185097 26737 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 17:06:12.185266 26737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.001s sys 0.000s
I20260806 17:06:12.185674 26801 tablet_bootstrap.cc:492] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588: Bootstrap starting.
I20260806 17:06:12.201958 26737 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:40329
I20260806 17:06:12.202248 26910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:40329 every 8 connection(s)
I20260806 17:06:12.203208 26737 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:06:12.205084 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 26737
I20260806 17:06:12.205639 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:34373
--local_ip_for_outbound_sockets=127.9.111.66
--tserver_master_addrs=127.9.111.126:34923
--webserver_port=33437
--webserver_interface=127.9.111.66
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:06:12.216959 26801 log.cc:824] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:12.225800 26801 tablet_bootstrap.cc:492] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 17:06:12.226359 26801 tablet_bootstrap.cc:492] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588: Bootstrap complete.
I20260806 17:06:12.227542 26801 ts_tablet_manager.cc:1397] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588: Time spent bootstrapping tablet: real 0.042s user 0.032s sys 0.007s
I20260806 17:06:12.233055 26801 raft_consensus.cc:348] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } }
I20260806 17:06:12.233686 26801 raft_consensus.cc:729] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82a0d759729741da91b829b16df9b588, State: Initialized, Role: FOLLOWER
I20260806 17:06:12.234385 26801 consensus_queue.cc:260] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } }
I20260806 17:06:12.235939 26801 ts_tablet_manager.cc:1428] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
W20260806 17:06:12.298863 26913 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:06:12.205743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:44125)
0806 17:06:12.207510 (+ 1767us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:12.207588 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:12.209791 (+ 2203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:12.296937 (+ 87146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1467,"thread_start_us":168,"threads_started":1}
W20260806 17:06:12.299546 26911 heartbeater.cc:644] Failed to heartbeat to 127.9.111.126:34923 (0 consecutive failures): Network error: Failed to ping master at 127.9.111.126:34923: Client connection negotiation failed: client connection to 127.9.111.126:34923: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 17:06:12.301081 26913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:12.300363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:57693)
0806 17:06:12.300542 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:12.300810 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 17:06:12.302940 26913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:12.302432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:43823)
0806 17:06:12.302578 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:12.302859 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 17:06:12.303210 26911 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 17:06:12.388958 26914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:12.389997 26914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:12.390813 26914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:12.398808 26914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:12.398898 26914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:12.398969 26914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:06:12.405435 26914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:34373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=33437
--tserver_master_addrs=127.9.111.126:34923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:12.407043 26914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:12.409567 26914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:12.423986 26923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:12.424045 26922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:12.424045 26925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:12.425407 26914 server_base.cc:1034] running on GCE node
I20260806 17:06:12.426070 26914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:12.426863 26914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:12.428081 26914 hybrid_clock.cc:648] HybridClock initialized: now 1786035972428064 us; error 42 us; skew 500 ppm
Aug 06 17:06:12 dist-test-slave-69rg krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035972, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:12.432293 26914 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:06:12.434386 26914 webserver.cc:460] Webserver started at http://127.9.111.66:33437/ using document root <none> and password file <none>
I20260806 17:06:12.435096 26914 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:12.435215 26914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:12.440478 26914 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 17:06:12.443274 26932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:12.445283 26914 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 17:06:12.445425 26914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "6cadb4cd97ae44f89701819e2acb73b0"
format_stamp: "Formatted at 2026-08-06 17:05:30 on dist-test-slave-69rg"
server_key: "052df19c75270527c8d4758305f62030"
server_key_iv: "642cdf39d5eacfc67dacbd5d9f503421"
server_key_version: "encryptionkey@0"
I20260806 17:06:12.446148 26914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:12.464027 26914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:12.467844 26914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:12.468030 26914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:12.468673 26914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:12.469551 26914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:12.469606 26914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:12.469650 26914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:12.469666 26914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:12.483601 26914 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:34373
I20260806 17:06:12.483659 27046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:34373 every 8 connection(s)
I20260806 17:06:12.484791 26914 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:06:12.489461 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:12.487416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:58731)
0806 17:06:12.488235 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:12.488359 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":151,"threads_started":1}
W20260806 17:06:12.490322 27047 heartbeater.cc:644] Failed to heartbeat to 127.9.111.126:34923 (0 consecutive failures): Network error: Failed to ping master at 127.9.111.126:34923: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
I20260806 17:06:12.491303 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:12.490902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:43107)
0806 17:06:12.491132 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:12.491216 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260806 17:06:12.491335 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 26914
I20260806 17:06:12.491850 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:43757
--local_ip_for_outbound_sockets=127.9.111.67
--tserver_master_addrs=127.9.111.126:34923
--webserver_port=36881
--webserver_interface=127.9.111.67
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:06:12.492265 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:12.491905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:33379)
0806 17:06:12.492156 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:12.492208 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
W20260806 17:06:12.492472 27047 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 17:06:12.631738 27050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:12.632090 27050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:12.632140 27050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:12.638149 27050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:12.638244 27050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:12.638520 27050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:06:12.645241 27050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:43757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=36881
--tserver_master_addrs=127.9.111.126:34923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:12.646588 27050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:12.647728 27050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:12.660197 27055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:12.660252 27056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:12.660252 27050 server_base.cc:1034] running on GCE node
W20260806 17:06:12.660588 27058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:12.661075 27050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:12.661969 27050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:12.663190 27050 hybrid_clock.cc:648] HybridClock initialized: now 1786035972663158 us; error 47 us; skew 500 ppm
Aug 06 17:06:12 dist-test-slave-69rg krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035972, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:12.666643 27050 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:06:12.667915 27050 webserver.cc:460] Webserver started at http://127.9.111.67:36881/ using document root <none> and password file <none>
I20260806 17:06:12.668865 27050 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:12.668968 27050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:12.675275 27050 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:06:12.678412 27065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:12.679956 27050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 17:06:12.680096 27050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "da07027f963c4af49c6472087d25f700"
format_stamp: "Formatted at 2026-08-06 17:05:30 on dist-test-slave-69rg"
server_key: "9768aee8ef24d9f869aaae47f0f6ab31"
server_key_iv: "2d66d1c2e23f1362c8481c3c1834c1ed"
server_key_version: "encryptionkey@0"
I20260806 17:06:12.680779 27050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:12.720207 27050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:12.724731 27050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:12.725003 27050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:12.725718 27050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:12.726679 27050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:12.726764 27050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:12.726841 27050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:12.726892 27050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:12.739512 27050 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:43757
I20260806 17:06:12.739588 27180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:43757 every 8 connection(s)
I20260806 17:06:12.741187 27050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:06:12.743521 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 27050
I20260806 17:06:12.746186 27184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:12.744334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.67:47181)
0806 17:06:12.745221 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:12.745344 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":175,"threads_started":1}
W20260806 17:06:12.746742 27181 heartbeater.cc:644] Failed to heartbeat to 127.9.111.126:34923 (0 consecutive failures): Network error: Failed to ping master at 127.9.111.126:34923: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
I20260806 17:06:12.747421 27184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:12.747180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.67:43937)
0806 17:06:12.747307 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:12.747363 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 17:06:12.748287 27184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:12.748030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.67:50223)
0806 17:06:12.748171 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:12.748223 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260806 17:06:12.748528 27181 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 17:06:13.304839 27185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:13.304084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:48007)
0806 17:06:13.304517 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:13.304590 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":118,"threads_started":1}
I20260806 17:06:13.493759 27186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:13.493223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:47043)
0806 17:06:13.493545 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:13.493636 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":92,"threads_started":1}
I20260806 17:06:13.734754 27187 raft_consensus.cc:482] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:06:13.735019 27187 raft_consensus.cc:504] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } }
I20260806 17:06:13.736187 27187 leader_election.cc:288] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers da07027f963c4af49c6472087d25f700 (127.9.111.67:43757), 6cadb4cd97ae44f89701819e2acb73b0 (127.9.111.66:34373)
I20260806 17:06:13.749771 27191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:13.749185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.67:37257)
0806 17:06:13.749557 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:13.749637 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":102,"threads_started":1}
Aug 06 17:06:13 dist-test-slave-69rg krb5kdc[25549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035972, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.67@KRBTEST.COM
Aug 06 17:06:13 dist-test-slave-69rg krb5kdc[25549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035972, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.66@KRBTEST.COM
I20260806 17:06:13.760385 27185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:13.736526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:43757 (local address 127.9.111.65:49167)
0806 17:06:13.736689 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:13.736706 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:13.736814 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:13.739187 (+ 2373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:13.739194 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:13.739876 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:13.740430 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:13.740455 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:13.743136 (+ 2681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:13.743140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:13.744011 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:13.744023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:13.747451 (+ 3428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:13.747463 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:13.747722 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:13.748972 (+ 1250us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:06:13.749010 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:13.751294 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:13.754759 (+ 3465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:13.754767 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:13.754786 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:13.755335 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:13.755916 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:13.755920 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:13.755922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:13.756189 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:13.757205 (+ 1016us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:13.757213 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:13.757611 (+ 398us) client_negotiation.cc:769] Sending connection context
0806 17:06:13.757904 (+ 293us) client_negotiation.cc:240] Negotiation successful
0806 17:06:13.758951 (+ 1047us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":241,"spinlock_wait_cycles":63488}
I20260806 17:06:13.760775 27190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:13.737160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:49167 (local address 127.9.111.67:43757)
0806 17:06:13.737575 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:13.737588 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:13.737631 (+ 43us) server_negotiation.cc:407] Connection header received
0806 17:06:13.738344 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:13.738352 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:13.738732 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:13.739748 (+ 1016us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:13.740690 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:13.742747 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:13.744179 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:13.746344 (+ 2165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:13.746568 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:13.751669 (+ 5101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:13.751678 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:13.751701 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:13.751859 (+ 158us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:13.754526 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:13.755530 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:13.755538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:13.755544 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:13.755737 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:13.756339 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:13.756343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:13.756345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:13.756998 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:13.757142 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:13.758382 (+ 1240us) server_negotiation.cc:299] Negotiation successful
0806 17:06:13.759964 (+ 1582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":87,"threads_started":1}
I20260806 17:06:13.760838 27189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:13.737035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:57503 (local address 127.9.111.66:34373)
0806 17:06:13.737476 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:13.737506 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:13.737561 (+ 55us) server_negotiation.cc:407] Connection header received
0806 17:06:13.738359 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:13.738366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:13.738731 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:13.739129 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:13.740759 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:13.742747 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:13.744281 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:13.746356 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:13.746589 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:13.751956 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:13.751961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:13.751973 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:13.752009 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:13.754526 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:13.755530 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:13.755538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:13.755544 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:13.755737 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:13.756339 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:13.756343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:13.756345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:13.756998 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:13.757153 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:13.758733 (+ 1580us) server_negotiation.cc:299] Negotiation successful
0806 17:06:13.759964 (+ 1231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":72,"threads_started":1}
I20260806 17:06:13.760390 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:13.736845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:34373 (local address 127.9.111.65:57503)
0806 17:06:13.737190 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:13.737206 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:13.737315 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:13.739187 (+ 1872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:13.739194 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:13.739862 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:13.740511 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:13.740523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:13.742939 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:13.742943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:13.744131 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:13.744143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:13.748269 (+ 4126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:13.748279 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:13.748385 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:13.749023 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:06:13.749039 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:13.751660 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:13.754759 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:13.754767 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:13.754786 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:13.755335 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:13.755916 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:13.755920 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:13.755922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:13.756189 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:13.757205 (+ 1016us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:13.757213 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:13.757642 (+ 429us) client_negotiation.cc:769] Sending connection context
0806 17:06:13.757904 (+ 262us) client_negotiation.cc:240] Negotiation successful
0806 17:06:13.759036 (+ 1132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":538,"spinlock_wait_cycles":19712,"thread_start_us":79,"threads_started":1}
W20260806 17:06:13.769068 26798 leader_election.cc:341] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da07027f963c4af49c6472087d25f700 (127.9.111.67:43757): Not found: Tablet not found: 77ca99084038479db0d7d3677562d6d8
W20260806 17:06:13.769323 26798 leader_election.cc:341] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6cadb4cd97ae44f89701819e2acb73b0 (127.9.111.66:34373): Not found: Tablet not found: 77ca99084038479db0d7d3677562d6d8
I20260806 17:06:13.769369 26798 leader_election.cc:302] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82a0d759729741da91b829b16df9b588; no voters: 6cadb4cd97ae44f89701819e2acb73b0, da07027f963c4af49c6472087d25f700
I20260806 17:06:13.769637 27187 raft_consensus.cc:2726] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 17:06:14.306619 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:14.306083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:46699)
0806 17:06:14.306447 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:14.306515 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":120,"threads_started":1}
I20260806 17:06:14.495440 27193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:14.494905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:38479)
0806 17:06:14.495255 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:14.495323 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260806 17:06:14.751456 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:14.750866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.67:41995)
0806 17:06:14.751253 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:14.751334 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":93,"threads_started":1}
I20260806 17:06:15.308578 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:15.307941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:49513)
0806 17:06:15.308385 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:15.308455 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":112,"threads_started":1}
I20260806 17:06:15.496999 27196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:15.496514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:50091)
0806 17:06:15.496816 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:15.496878 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":84,"threads_started":1}
I20260806 17:06:15.654343 27197 raft_consensus.cc:482] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:06:15.654531 27197 raft_consensus.cc:504] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } }
I20260806 17:06:15.655139 27197 leader_election.cc:288] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers da07027f963c4af49c6472087d25f700 (127.9.111.67:43757), 6cadb4cd97ae44f89701819e2acb73b0 (127.9.111.66:34373)
W20260806 17:06:15.656162 26798 leader_election.cc:341] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da07027f963c4af49c6472087d25f700 (127.9.111.67:43757): Not found: Tablet not found: 77ca99084038479db0d7d3677562d6d8
W20260806 17:06:15.656354 26798 leader_election.cc:341] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6cadb4cd97ae44f89701819e2acb73b0 (127.9.111.66:34373): Not found: Tablet not found: 77ca99084038479db0d7d3677562d6d8
I20260806 17:06:15.656420 26798 leader_election.cc:302] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82a0d759729741da91b829b16df9b588; no voters: 6cadb4cd97ae44f89701819e2acb73b0, da07027f963c4af49c6472087d25f700
I20260806 17:06:15.656582 27197 raft_consensus.cc:2726] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 17:06:15.752959 27198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:15.752409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.67:59825)
0806 17:06:15.752772 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:15.752839 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":108,"threads_started":1}
I20260806 17:06:16.310199 27199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:16.309597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:55263)
0806 17:06:16.309995 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:16.310060 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":133,"threads_started":1}
I20260806 17:06:16.498477 27200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:16.497995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:34115)
0806 17:06:16.498298 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:16.498365 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":99,"threads_started":1}
I20260806 17:06:16.754451 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:16.754025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.67:58573)
0806 17:06:16.754295 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:16.754355 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
I20260806 17:06:17.311978 27202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:17.311409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:49479)
0806 17:06:17.311801 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:17.311874 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":85,"threads_started":1}
I20260806 17:06:17.499939 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:17.499456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:37327)
0806 17:06:17.499774 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:17.499840 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":101,"threads_started":1}
I20260806 17:06:17.756050 27204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:17.755553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.67:58031)
0806 17:06:17.755876 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:17.755943 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":116,"threads_started":1}
I20260806 17:06:18.050597 27205 raft_consensus.cc:482] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:06:18.050803 27205 raft_consensus.cc:504] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } }
I20260806 17:06:18.051380 27205 leader_election.cc:288] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers da07027f963c4af49c6472087d25f700 (127.9.111.67:43757), 6cadb4cd97ae44f89701819e2acb73b0 (127.9.111.66:34373)
W20260806 17:06:18.052529 26798 leader_election.cc:341] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da07027f963c4af49c6472087d25f700 (127.9.111.67:43757): Not found: Tablet not found: 77ca99084038479db0d7d3677562d6d8
W20260806 17:06:18.052670 26798 leader_election.cc:341] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6cadb4cd97ae44f89701819e2acb73b0 (127.9.111.66:34373): Not found: Tablet not found: 77ca99084038479db0d7d3677562d6d8
I20260806 17:06:18.052709 26798 leader_election.cc:302] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82a0d759729741da91b829b16df9b588; no voters: 6cadb4cd97ae44f89701819e2acb73b0, da07027f963c4af49c6472087d25f700
I20260806 17:06:18.052855 27205 raft_consensus.cc:2726] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 17:06:18.313536 27206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:18.313079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:45783)
0806 17:06:18.313383 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:18.313439 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":114,"threads_started":1}
I20260806 17:06:18.501475 27207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:18.500951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:39581)
0806 17:06:18.501286 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:18.501360 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
I20260806 17:06:18.757869 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:18.757253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.67:50829)
0806 17:06:18.757641 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:18.757705 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":203,"threads_started":1}
I20260806 17:06:19.315234 27209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:19.314629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:41759)
0806 17:06:19.315043 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:19.315119 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":87,"threads_started":1}
I20260806 17:06:19.503108 27210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:19.502525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:58517)
0806 17:06:19.502930 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:19.503004 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":97,"threads_started":1}
I20260806 17:06:19.759716 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:19.759011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.67:33699)
0806 17:06:19.759529 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:19.759601 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":92,"threads_started":1}
I20260806 17:06:20.316610 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:20.316155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:56531)
0806 17:06:20.316435 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:20.316506 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":82,"threads_started":1}
I20260806 17:06:20.504900 27213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:20.504201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:48487)
0806 17:06:20.504632 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:20.504719 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":123,"threads_started":1}
I20260806 17:06:20.761273 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:20.760792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.67:38715)
0806 17:06:20.761107 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:20.761173 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":111,"threads_started":1}
I20260806 17:06:21.318368 27215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:21.317646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:56179)
0806 17:06:21.318144 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:21.318223 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":112,"threads_started":1}
I20260806 17:06:21.506528 27216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:21.506010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:34545)
0806 17:06:21.506369 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:21.506429 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":115,"threads_started":1}
I20260806 17:06:21.762992 27217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:21.762490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.67:51699)
0806 17:06:21.762824 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:21.762892 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":92,"threads_started":1}
I20260806 17:06:22.320048 27218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:22.319498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:58617)
0806 17:06:22.319865 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:22.319941 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":107,"threads_started":1}
I20260806 17:06:22.508080 27219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:22.507568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:41127)
0806 17:06:22.507904 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:22.507976 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":117,"threads_started":1}
I20260806 17:06:22.522416 27220 raft_consensus.cc:482] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:06:22.522559 27220 raft_consensus.cc:504] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } }
I20260806 17:06:22.523028 27220 leader_election.cc:288] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers da07027f963c4af49c6472087d25f700 (127.9.111.67:43757), 6cadb4cd97ae44f89701819e2acb73b0 (127.9.111.66:34373)
W20260806 17:06:22.524255 26798 leader_election.cc:341] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da07027f963c4af49c6472087d25f700 (127.9.111.67:43757): Not found: Tablet not found: 77ca99084038479db0d7d3677562d6d8
W20260806 17:06:22.524439 26798 leader_election.cc:341] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6cadb4cd97ae44f89701819e2acb73b0 (127.9.111.66:34373): Not found: Tablet not found: 77ca99084038479db0d7d3677562d6d8
I20260806 17:06:22.524475 26798 leader_election.cc:302] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82a0d759729741da91b829b16df9b588; no voters: 6cadb4cd97ae44f89701819e2acb73b0, da07027f963c4af49c6472087d25f700
I20260806 17:06:22.524698 27220 raft_consensus.cc:2726] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 17:06:22.764622 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:22.764114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.67:57257)
0806 17:06:22.764449 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:22.764519 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":102,"threads_started":1}
I20260806 17:06:23.321484 27222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:23.320964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:33155)
0806 17:06:23.321292 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:23.321360 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":87,"threads_started":1}
I20260806 17:06:23.509641 27223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:23.509144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:47197)
0806 17:06:23.509452 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:23.509535 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":106,"threads_started":1}
I20260806 17:06:23.766294 27225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:23.765756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.67:46371)
0806 17:06:23.766102 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:23.766168 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":154,"threads_started":1}
I20260806 17:06:24.322973 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:24.322508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:57923)
0806 17:06:24.322797 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:24.322866 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":83,"threads_started":1}
I20260806 17:06:24.511338 27227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:24.510783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:48757)
0806 17:06:24.511102 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:24.511184 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":96,"threads_started":1}
I20260806 17:06:24.767596 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:24.767170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.67:55789)
0806 17:06:24.767440 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:24.767498 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
I20260806 17:06:25.324683 27229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:25.324086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:37589)
0806 17:06:25.324488 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:25.324581 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":186,"threads_started":1}
I20260806 17:06:25.512939 27230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:25.512426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:56405)
0806 17:06:25.512766 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:25.512837 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":86,"threads_started":1}
I20260806 17:06:25.768994 27231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:25.768516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.67:44723)
0806 17:06:25.768828 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:25.768897 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":106,"threads_started":1}
I20260806 17:06:26.326236 27232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:26.325704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:39657)
0806 17:06:26.326060 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:26.326119 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":150,"threads_started":1}
I20260806 17:06:26.484222 27233 raft_consensus.cc:482] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:06:26.484357 27233 raft_consensus.cc:504] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da07027f963c4af49c6472087d25f700" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 43757 } } peers { permanent_uuid: "82a0d759729741da91b829b16df9b588" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40329 } } peers { permanent_uuid: "6cadb4cd97ae44f89701819e2acb73b0" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34373 } }
I20260806 17:06:26.484876 27233 leader_election.cc:288] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers da07027f963c4af49c6472087d25f700 (127.9.111.67:43757), 6cadb4cd97ae44f89701819e2acb73b0 (127.9.111.66:34373)
W20260806 17:06:26.485878 26798 leader_election.cc:341] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da07027f963c4af49c6472087d25f700 (127.9.111.67:43757): Not found: Tablet not found: 77ca99084038479db0d7d3677562d6d8
W20260806 17:06:26.486042 26798 leader_election.cc:341] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6cadb4cd97ae44f89701819e2acb73b0 (127.9.111.66:34373): Not found: Tablet not found: 77ca99084038479db0d7d3677562d6d8
I20260806 17:06:26.486097 26798 leader_election.cc:302] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82a0d759729741da91b829b16df9b588; no voters: 6cadb4cd97ae44f89701819e2acb73b0, da07027f963c4af49c6472087d25f700
I20260806 17:06:26.486275 27233 raft_consensus.cc:2726] T 77ca99084038479db0d7d3677562d6d8 P 82a0d759729741da91b829b16df9b588 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 17:06:26.514619 27234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:26.514135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:52997)
0806 17:06:26.514452 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:26.514519 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":90,"threads_started":1}
I20260806 17:06:26.770540 27235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:26.770012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.67:50979)
0806 17:06:26.770351 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:26.770417 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":98,"threads_started":1}
I20260806 17:06:27.327836 27236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:27.327422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.65:58671)
0806 17:06:27.327680 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:27.327735 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260806 17:06:27.516744 27237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:27.516191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:34923 (local address 127.9.111.66:34229)
0806 17:06:27.516584 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:27.516645 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.126:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":106,"threads_started":1}
I20260806 17:06:27.750675 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 26737
I20260806 17:06:27.758149 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 26914
I20260806 17:06:27.765368 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 27050
2026-08-06T17:06:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59616 ms)
[----------] 30 tests from SecurityITest (254880 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:06:27 dist-test-slave-69rg krb5kdc[27240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:06:27 dist-test-slave-69rg krb5kdc[27240](info): set up 2 sockets
Aug 06 17:06:27 dist-test-slave-69rg krb5kdc[27240](info): commencing operation
krb5kdc: starting...
W20260806 17:06:28.837426 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:06:28 dist-test-slave-69rg krb5kdc[27240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:06:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:06:28.981261 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:38169
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:38169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:29.098464 27253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:29.098767 27253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:29.098810 27253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:29.103091 27253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:29.103284 27253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:29.103344 27253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:29.103416 27253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:29.103456 27253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:29.112713 27253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:38169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:38169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:29.113982 27253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:29.115027 27253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 17:06:29.124218 27253 server_base.cc:1034] running on GCE node
W20260806 17:06:29.124130 27258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:29.124132 27259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:29.124137 27261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:29.125099 27253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:29.126348 27253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:29.127534 27253 hybrid_clock.cc:648] HybridClock initialized: now 1786035989127504 us; error 47 us; skew 500 ppm
Aug 06 17:06:29 dist-test-slave-69rg krb5kdc[27240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035989, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:29.130785 27253 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:06:29.131783 27253 webserver.cc:460] Webserver started at http://127.9.111.126:40995/ using document root <none> and password file <none>
I20260806 17:06:29.132354 27253 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:29.132417 27253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:29.132596 27253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:29.134862 27253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "9dce969061c5410cbbe7ebdf75693321"
format_stamp: "Formatted at 2026-08-06 17:06:29 on dist-test-slave-69rg"
server_key: "c11c5b20ee541df5f83e69e6b3fdf4de"
server_key_iv: "3a97dc2d8240a84e0ff10c44e81eb7e9"
server_key_version: "encryptionkey@0"
I20260806 17:06:29.135437 27253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "9dce969061c5410cbbe7ebdf75693321"
format_stamp: "Formatted at 2026-08-06 17:06:29 on dist-test-slave-69rg"
server_key: "c11c5b20ee541df5f83e69e6b3fdf4de"
server_key_iv: "3a97dc2d8240a84e0ff10c44e81eb7e9"
server_key_version: "encryptionkey@0"
I20260806 17:06:29.139730 27253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 17:06:29.142519 27268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:29.143844 27253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:06:29.143957 27253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "9dce969061c5410cbbe7ebdf75693321"
format_stamp: "Formatted at 2026-08-06 17:06:29 on dist-test-slave-69rg"
server_key: "c11c5b20ee541df5f83e69e6b3fdf4de"
server_key_iv: "3a97dc2d8240a84e0ff10c44e81eb7e9"
server_key_version: "encryptionkey@0"
I20260806 17:06:29.144052 27253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:29.162297 27253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:29.165642 27253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:29.165889 27253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:29.173907 27253 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:38169
I20260806 17:06:29.173941 27320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:38169 every 8 connection(s)
I20260806 17:06:29.174912 27253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:29.177800 27321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:29.179437 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 27253
I20260806 17:06:29.179574 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:29.179910 9661 external_mini_cluster.cc:1428] Setting key eb36710ac47e37dfd21443cc99d7def4
I20260806 17:06:29.184935 27321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321: Bootstrap starting.
I20260806 17:06:29.187266 27321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:29.188062 27321 log.cc:824] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:29.190392 27321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321: No bootstrap required, opened a new log
Aug 06 17:06:29 dist-test-slave-69rg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:06:29.194308 27321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dce969061c5410cbbe7ebdf75693321" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38169 } }
I20260806 17:06:29.194574 27321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:29.194664 27321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dce969061c5410cbbe7ebdf75693321, State: Initialized, Role: FOLLOWER
I20260806 17:06:29.195185 27321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dce969061c5410cbbe7ebdf75693321" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38169 } }
I20260806 17:06:29.195384 27321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:29.195477 27321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:29.195593 27321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:29.196856 27321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dce969061c5410cbbe7ebdf75693321" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38169 } }
I20260806 17:06:29.197218 27321 leader_election.cc:302] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dce969061c5410cbbe7ebdf75693321; no voters:
I20260806 17:06:29.197551 27321 leader_election.cc:288] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:29.197726 27327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:29.198025 27325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:29.181406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.9.111.126:38169)
0806 17:06:29.181936 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:29.181958 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:29.183007 (+ 1049us) server_negotiation.cc:407] Connection header received
0806 17:06:29.184097 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:29.184117 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:29.184499 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:29.184877 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:29.185847 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.187053 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.188111 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.188902 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.189074 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:29.192080 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:29.192087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:29.192102 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:29.192133 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:29.194162 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:29.194706 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:29.194715 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:29.194722 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:29.194890 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:29.195279 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:29.195282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:29.195284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:29.195883 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:29.196051 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:29.196421 (+ 370us) server_negotiation.cc:299] Negotiation successful
0806 17:06:29.196835 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":152,"threads_started":1}
I20260806 17:06:29.198027 27327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321 [term 1 LEADER]: Becoming Leader. State: Replica: 9dce969061c5410cbbe7ebdf75693321, State: Running, Role: LEADER
I20260806 17:06:29.198627 27327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dce969061c5410cbbe7ebdf75693321" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38169 } }
I20260806 17:06:29.199015 27321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:29.201675 27328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dce969061c5410cbbe7ebdf75693321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dce969061c5410cbbe7ebdf75693321" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38169 } } }
I20260806 17:06:29.201720 27329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dce969061c5410cbbe7ebdf75693321. Latest consensus state: current_term: 1 leader_uuid: "9dce969061c5410cbbe7ebdf75693321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dce969061c5410cbbe7ebdf75693321" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38169 } } }
I20260806 17:06:29.201876 27328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:29.201874 27329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321 [sys.catalog]: This master's current role is: LEADER
W20260806 17:06:29.204550 27341 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:06:29.204658 27341 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:06:29.204753 27342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:29.205590 27342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:29.213096 27342 catalog_manager.cc:1347] Generated new cluster ID: 74a7022fae3140d4964c0fba2db7964d
I20260806 17:06:29.213204 27342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:29.219234 27342 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:29.220366 27342 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:29.229423 27342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321: Generated new TSK 0
I20260806 17:06:29.230338 27342 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
I20260806 17:06:29.267086 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38169
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:29.386919 27348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:29.387239 27348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:29.387285 27348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:29.391616 27348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:29.391695 27348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:29.391788 27348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:06:29.397508 27348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:29.398748 27348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:29.399964 27348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:29.411022 27356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:29.411029 27354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:29.411069 27348 server_base.cc:1034] running on GCE node
W20260806 17:06:29.411028 27353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:29.411692 27348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:29.412354 27348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:29.413571 27348 hybrid_clock.cc:648] HybridClock initialized: now 1786035989413540 us; error 41 us; skew 500 ppm
Aug 06 17:06:29 dist-test-slave-69rg krb5kdc[27240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035989, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:29.421732 27348 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:06:29.422771 27348 webserver.cc:460] Webserver started at http://127.9.111.65:43173/ using document root <none> and password file <none>
I20260806 17:06:29.423316 27348 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:29.423375 27348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:29.423547 27348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:29.425575 27348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "ea9430b6ad3f4203bf91e28f032217eb"
format_stamp: "Formatted at 2026-08-06 17:06:29 on dist-test-slave-69rg"
server_key: "603876b8acce05a715e1840cfd72f639"
server_key_iv: "28f26546c99598979fa9873b7c214b55"
server_key_version: "encryptionkey@0"
I20260806 17:06:29.426183 27348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "ea9430b6ad3f4203bf91e28f032217eb"
format_stamp: "Formatted at 2026-08-06 17:06:29 on dist-test-slave-69rg"
server_key: "603876b8acce05a715e1840cfd72f639"
server_key_iv: "28f26546c99598979fa9873b7c214b55"
server_key_version: "encryptionkey@0"
I20260806 17:06:29.430094 27348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:06:29.432623 27364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:29.434338 27348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:06:29.434458 27348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "ea9430b6ad3f4203bf91e28f032217eb"
format_stamp: "Formatted at 2026-08-06 17:06:29 on dist-test-slave-69rg"
server_key: "603876b8acce05a715e1840cfd72f639"
server_key_iv: "28f26546c99598979fa9873b7c214b55"
server_key_version: "encryptionkey@0"
I20260806 17:06:29.434559 27348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:29.448642 27348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:29.452402 27348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:29.452656 27348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:29.453346 27348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:29.454329 27348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:29.454376 27348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:29.454445 27348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:29.454478 27348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:29.463485 27348 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:42463
I20260806 17:06:29.463543 27478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:42463 every 8 connection(s)
I20260806 17:06:29.464493 27348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:06:29.465322 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 27348
I20260806 17:06:29.465493 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:06:29.465899 9661 external_mini_cluster.cc:1428] Setting key 4a125c9286e42f8d3fcbae26d758dc13
Aug 06 17:06:29 dist-test-slave-69rg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035989, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.111.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.66@KRBTEST.COM" created.
I20260806 17:06:29.480114 27325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:29.466734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:55201 (local address 127.9.111.126:38169)
0806 17:06:29.466889 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:29.466893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:29.467665 (+ 772us) server_negotiation.cc:407] Connection header received
0806 17:06:29.469002 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:29.469009 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:29.469075 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:29.469180 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:29.470893 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.471759 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.473028 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.473479 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.473612 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:29.476268 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:29.476276 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:29.476282 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:29.476319 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:29.477888 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:29.478458 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:29.478461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:29.478462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:29.478539 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:29.478869 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:29.478871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:29.478873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:29.479054 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:29.479131 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:29.479674 (+ 543us) server_negotiation.cc:299] Negotiation successful
0806 17:06:29.479850 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 17:06:29.481045 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:29.467010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38169 (local address 127.9.111.65:55201)
0806 17:06:29.467491 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:29.467546 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:29.468783 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:29.469366 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:29.469378 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:29.470000 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:29.470717 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:29.470739 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.471896 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.471898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:29.472879 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:29.472889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.473652 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.473655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:29.473868 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:29.474553 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:06:29.474569 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:29.476062 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:29.478033 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:29.478042 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:29.478060 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:29.478363 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:29.478624 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:29.478626 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:29.478628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:29.478779 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:29.479142 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:29.479150 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:29.479419 (+ 269us) client_negotiation.cc:769] Sending connection context
0806 17:06:29.479609 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 17:06:29.479995 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":154,"threads_started":1}
I20260806 17:06:29.482350 27479 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38169
I20260806 17:06:29.482617 27479 heartbeater.cc:464] Registering TS with master...
I20260806 17:06:29.483160 27479 heartbeater.cc:511] Master 127.9.111.126:38169 requested a full tablet report, sending...
I20260806 17:06:29.484894 27285 ts_manager.cc:194] Registered new tserver with Master: ea9430b6ad3f4203bf91e28f032217eb (127.9.111.65:42463)
I20260806 17:06:29.486704 27285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:55201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
I20260806 17:06:29.502144 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38169
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:29.616328 27484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:29.616639 27484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:29.616683 27484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:29.621003 27484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:29.621094 27484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:29.621156 27484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:06:29.626847 27484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:29.628012 27484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:29.629091 27484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:29.639122 27492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:29.639117 27490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:29.639122 27489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:29.639442 27484 server_base.cc:1034] running on GCE node
I20260806 17:06:29.639916 27484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:29.640539 27484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:29.641796 27484 hybrid_clock.cc:648] HybridClock initialized: now 1786035989641749 us; error 29 us; skew 500 ppm
Aug 06 17:06:29 dist-test-slave-69rg krb5kdc[27240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035989, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:29.644846 27484 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:06:29.645856 27484 webserver.cc:460] Webserver started at http://127.9.111.66:35463/ using document root <none> and password file <none>
I20260806 17:06:29.646464 27484 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:29.646528 27484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:29.646811 27484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:29.649096 27484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "a5f6744d1986420f8ef2ecdab790fa7b"
format_stamp: "Formatted at 2026-08-06 17:06:29 on dist-test-slave-69rg"
server_key: "993e94da1939686efbbaaf33355f0b34"
server_key_iv: "20b64c6ff016938f3d24fb49abb7c0f3"
server_key_version: "encryptionkey@0"
I20260806 17:06:29.649647 27484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "a5f6744d1986420f8ef2ecdab790fa7b"
format_stamp: "Formatted at 2026-08-06 17:06:29 on dist-test-slave-69rg"
server_key: "993e94da1939686efbbaaf33355f0b34"
server_key_iv: "20b64c6ff016938f3d24fb49abb7c0f3"
server_key_version: "encryptionkey@0"
I20260806 17:06:29.653483 27484 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260806 17:06:29.655944 27499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:29.657150 27484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 17:06:29.657284 27484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "a5f6744d1986420f8ef2ecdab790fa7b"
format_stamp: "Formatted at 2026-08-06 17:06:29 on dist-test-slave-69rg"
server_key: "993e94da1939686efbbaaf33355f0b34"
server_key_iv: "20b64c6ff016938f3d24fb49abb7c0f3"
server_key_version: "encryptionkey@0"
I20260806 17:06:29.657394 27484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:29.673272 27484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:29.676730 27484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:29.676975 27484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:29.678020 27484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:29.678984 27484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:29.679039 27484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:29.679091 27484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:29.679152 27484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:29.689484 27484 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:33229
I20260806 17:06:29.689502 27615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:33229 every 8 connection(s)
I20260806 17:06:29.690542 27484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:06:29.700130 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 27484
I20260806 17:06:29.700279 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:06:29.700604 9661 external_mini_cluster.cc:1428] Setting key b314bef033134244d19085191f75211e
Aug 06 17:06:29 dist-test-slave-69rg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035989, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:06:29.706820 27325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:29.692787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:36559 (local address 127.9.111.126:38169)
0806 17:06:29.692969 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:29.692973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:29.693782 (+ 809us) server_negotiation.cc:407] Connection header received
0806 17:06:29.695239 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:29.695242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:29.695326 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:29.695412 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:29.696866 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.697505 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.698950 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.699277 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.699374 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:29.702643 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:29.702647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:29.702650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:29.702684 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:29.704242 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:29.704954 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:29.704957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:29.704959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:29.705051 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:29.705444 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:29.705447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:29.705449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:29.705661 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:29.705781 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:29.706544 (+ 763us) server_negotiation.cc:299] Negotiation successful
0806 17:06:29.706677 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 17:06:29.708153 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:29.693078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38169 (local address 127.9.111.66:36559)
0806 17:06:29.693601 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:29.693649 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:29.694985 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:29.695610 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:29.695618 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:29.696068 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:29.696606 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:29.696658 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.697663 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.697666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:29.698738 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:29.698750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.699446 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.699449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:29.699617 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:29.700441 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:06:29.700463 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:29.702460 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:29.704485 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:29.704492 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:29.704512 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:29.704831 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:29.705156 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:29.705159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:29.705162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:29.705336 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:29.705779 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:29.705789 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:29.706200 (+ 411us) client_negotiation.cc:769] Sending connection context
0806 17:06:29.706439 (+ 239us) client_negotiation.cc:240] Negotiation successful
0806 17:06:29.706851 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":140,"threads_started":1}
I20260806 17:06:29.709935 27616 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38169
I20260806 17:06:29.710234 27616 heartbeater.cc:464] Registering TS with master...
I20260806 17:06:29.710855 27616 heartbeater.cc:511] Master 127.9.111.126:38169 requested a full tablet report, sending...
I20260806 17:06:29.712139 27285 ts_manager.cc:194] Registered new tserver with Master: a5f6744d1986420f8ef2ecdab790fa7b (127.9.111.66:33229)
WARNING: no policy specified for kudu/127.9.111.67@KRBTEST.COM; defaulting to no policy
I20260806 17:06:29.712829 27285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:36559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
I20260806 17:06:29.739461 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38169
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:29.858033 27621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:29.858397 27621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:29.858494 27621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:29.862934 27621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:29.863039 27621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:29.863138 27621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:06:29.868726 27621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:29.869948 27621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:29.871025 27621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:29.881129 27627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:29.881196 27626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:29.881145 27629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:29.881302 27621 server_base.cc:1034] running on GCE node
I20260806 17:06:29.881919 27621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:29.882653 27621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:29.883867 27621 hybrid_clock.cc:648] HybridClock initialized: now 1786035989883849 us; error 29 us; skew 500 ppm
Aug 06 17:06:29 dist-test-slave-69rg krb5kdc[27240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035989, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:29.886986 27621 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:06:29.887987 27621 webserver.cc:460] Webserver started at http://127.9.111.67:35137/ using document root <none> and password file <none>
I20260806 17:06:29.888563 27621 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:29.888628 27621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:29.888859 27621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:29.891126 27621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "83c1d02f99674814866afaa7e2c14622"
format_stamp: "Formatted at 2026-08-06 17:06:29 on dist-test-slave-69rg"
server_key: "535ada448994cd1d76590559fa676d43"
server_key_iv: "8b7cbc6af629da10d67177c91fac9791"
server_key_version: "encryptionkey@0"
I20260806 17:06:29.891754 27621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "83c1d02f99674814866afaa7e2c14622"
format_stamp: "Formatted at 2026-08-06 17:06:29 on dist-test-slave-69rg"
server_key: "535ada448994cd1d76590559fa676d43"
server_key_iv: "8b7cbc6af629da10d67177c91fac9791"
server_key_version: "encryptionkey@0"
I20260806 17:06:29.895895 27621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 17:06:29.899317 27636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:29.900887 27621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 17:06:29.901038 27621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "83c1d02f99674814866afaa7e2c14622"
format_stamp: "Formatted at 2026-08-06 17:06:29 on dist-test-slave-69rg"
server_key: "535ada448994cd1d76590559fa676d43"
server_key_iv: "8b7cbc6af629da10d67177c91fac9791"
server_key_version: "encryptionkey@0"
I20260806 17:06:29.901177 27621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:29.918264 27621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:29.921511 27621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:29.921736 27621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:29.922390 27621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:29.923245 27621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:29.923321 27621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:29.923391 27621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:29.923442 27621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:29.932966 27621 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:37845
I20260806 17:06:29.932987 27752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:37845 every 8 connection(s)
I20260806 17:06:29.933946 27621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:06:29.938155 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 27621
I20260806 17:06:29.938338 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:06:29.938707 9661 external_mini_cluster.cc:1428] Setting key 7970f06ea3bee7375c732f73d04d4769
Aug 06 17:06:29 dist-test-slave-69rg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035989, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:06:29.949138 27325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:29.935843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:49775 (local address 127.9.111.126:38169)
0806 17:06:29.936017 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:29.936022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:29.936905 (+ 883us) server_negotiation.cc:407] Connection header received
0806 17:06:29.938660 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:29.938664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:29.938730 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:29.938820 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:29.940559 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.941263 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.942580 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.942798 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.942880 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:29.945429 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:29.945432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:29.945435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:29.945461 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:29.946963 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:29.947542 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:29.947546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:29.947548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:29.947660 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:29.947977 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:29.947980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:29.947981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:29.948244 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:29.948335 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:29.948889 (+ 554us) server_negotiation.cc:299] Negotiation successful
0806 17:06:29.948999 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 17:06:29.950065 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:29.936205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38169 (local address 127.9.111.67:49775)
0806 17:06:29.936738 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:29.936785 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:29.938300 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:29.939171 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:29.939183 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:29.939783 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:29.940411 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:29.940427 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.941385 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.941388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:29.942464 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:29.942475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.942958 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.942961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:29.943111 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:29.943692 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:06:29.943722 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:29.945259 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:29.947073 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:29.947078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:29.947091 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:29.947424 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:29.947750 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:29.947753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:29.947754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:29.947888 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:29.948346 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:29.948350 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:29.948629 (+ 279us) client_negotiation.cc:769] Sending connection context
0806 17:06:29.948812 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 17:06:29.949078 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":128,"threads_started":1}
I20260806 17:06:29.951299 27753 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38169
I20260806 17:06:29.951576 27753 heartbeater.cc:464] Registering TS with master...
I20260806 17:06:29.952175 27753 heartbeater.cc:511] Master 127.9.111.126:38169 requested a full tablet report, sending...
I20260806 17:06:29.953292 27285 ts_manager.cc:194] Registered new tserver with Master: 83c1d02f99674814866afaa7e2c14622 (127.9.111.67:37845)
I20260806 17:06:29.953969 27285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:49775
I20260806 17:06:29.965003 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 17:06:29.975340 27325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:29.967270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.9.111.126:38169)
0806 17:06:29.967419 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:29.967423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:29.967587 (+ 164us) server_negotiation.cc:407] Connection header received
0806 17:06:29.967679 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:29.967681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:29.967722 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:29.967794 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:29.968545 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.969058 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.970079 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.970339 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.970432 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:29.972042 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:29.972045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:29.972047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:29.972071 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:29.973655 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:29.974132 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:29.974135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:29.974136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:29.974262 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:29.974518 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:29.974520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:29.974522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:29.974766 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:29.974874 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:29.975079 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 17:06:29.975184 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 17:06:29.978555 27285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:06:29.981549 27285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:06:29.996404 27766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:29.990520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:33229 (local address 127.0.0.1:53290)
0806 17:06:29.991287 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:29.991342 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:29.991491 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:29.992500 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:29.992503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:29.992519 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:29.992662 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:29.992668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.993678 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.993683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:29.995004 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:29.995015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.995837 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.995840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:29.995953 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:29.995989 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 17:06:29.996116 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 17:06:29.996248 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":2,"spinlock_wait_cycles":384,"thread_start_us":116,"threads_started":1}
I20260806 17:06:29.996411 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:29.991041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53290 (local address 127.9.111.66:33229)
0806 17:06:29.991837 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:29.991849 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:29.991873 (+ 24us) server_negotiation.cc:407] Connection header received
0806 17:06:29.991948 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:29.991953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:29.992164 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:29.992323 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:29.992789 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.993536 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.995126 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.995704 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.995837 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:29.995997 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:29.996146 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 17:06:29.996246 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":132,"threads_started":1}
I20260806 17:06:29.996913 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:29.990523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:42463 (local address 127.0.0.1:40840)
0806 17:06:29.991651 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:29.991667 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:29.991745 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:29.992515 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:29.992518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:29.992533 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:29.992703 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:29.992720 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.994204 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.994207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:29.995249 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:29.995259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.996469 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.996473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:29.996596 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:29.996622 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 17:06:29.996692 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 17:06:29.996770 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"mutex_wait_us":852,"spinlock_wait_cycles":9984,"thread_start_us":100,"threads_started":1}
I20260806 17:06:29.996985 27765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:29.990633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40840 (local address 127.9.111.65:42463)
0806 17:06:29.991988 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:29.991994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:29.992013 (+ 19us) server_negotiation.cc:407] Connection header received
0806 17:06:29.992077 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:29.992081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:29.992247 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:29.992366 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:29.992848 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.994042 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.995372 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.996185 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.996290 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:29.996409 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:29.996733 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 17:06:29.996834 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1254,"thread_start_us":139,"threads_started":1}
I20260806 17:06:29.998322 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:29.990838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:37845 (local address 127.0.0.1:41408)
0806 17:06:29.991362 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:29.991376 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:29.991492 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:29.992323 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:29.992329 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:29.992351 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:29.992517 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:29.992524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.995396 (+ 2872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.995398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:29.996515 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:29.996523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.997884 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.997887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:29.997995 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:29.998019 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 17:06:29.998087 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 17:06:29.998181 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":1,"spinlock_wait_cycles":10112,"thread_start_us":95,"threads_started":1}
I20260806 17:06:29.998740 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:29.991370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41408 (local address 127.9.111.67:37845)
0806 17:06:29.991845 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:29.991858 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:29.991882 (+ 24us) server_negotiation.cc:407] Connection header received
0806 17:06:29.991960 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:29.991965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:29.992162 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:29.992330 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:29.992664 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.994004 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.996663 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:29.997545 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:29.997660 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:29.997788 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:29.998488 (+ 700us) server_negotiation.cc:299] Negotiation successful
0806 17:06:29.998588 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":153,"threads_started":1}
I20260806 17:06:30.000627 27548 tablet_service.cc:1467] Processing CreateTablet for tablet 589afb72b1a44f1dba4cad902f5a12f6 (DEFAULT_TABLE table=test-table [id=f333396cc6c640819bad15afb1581be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:06:30.000661 27412 tablet_service.cc:1467] Processing CreateTablet for tablet 589afb72b1a44f1dba4cad902f5a12f6 (DEFAULT_TABLE table=test-table [id=f333396cc6c640819bad15afb1581be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:06:30.001657 27548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 589afb72b1a44f1dba4cad902f5a12f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:30.001730 27412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 589afb72b1a44f1dba4cad902f5a12f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:30.001510 27685 tablet_service.cc:1467] Processing CreateTablet for tablet 589afb72b1a44f1dba4cad902f5a12f6 (DEFAULT_TABLE table=test-table [id=f333396cc6c640819bad15afb1581be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:06:30.002624 27685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 589afb72b1a44f1dba4cad902f5a12f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:30.008550 27770 tablet_bootstrap.cc:492] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b: Bootstrap starting.
I20260806 17:06:30.010135 27771 tablet_bootstrap.cc:492] T 589afb72b1a44f1dba4cad902f5a12f6 P ea9430b6ad3f4203bf91e28f032217eb: Bootstrap starting.
I20260806 17:06:30.011508 27770 tablet_bootstrap.cc:654] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:30.012022 27771 tablet_bootstrap.cc:654] T 589afb72b1a44f1dba4cad902f5a12f6 P ea9430b6ad3f4203bf91e28f032217eb: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:30.012357 27770 log.cc:824] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:30.012758 27771 log.cc:824] T 589afb72b1a44f1dba4cad902f5a12f6 P ea9430b6ad3f4203bf91e28f032217eb: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:30.013471 27772 tablet_bootstrap.cc:492] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622: Bootstrap starting.
I20260806 17:06:30.015096 27770 tablet_bootstrap.cc:492] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b: No bootstrap required, opened a new log
I20260806 17:06:30.015372 27770 ts_tablet_manager.cc:1397] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 17:06:30.015401 27771 tablet_bootstrap.cc:492] T 589afb72b1a44f1dba4cad902f5a12f6 P ea9430b6ad3f4203bf91e28f032217eb: No bootstrap required, opened a new log
I20260806 17:06:30.015601 27771 ts_tablet_manager.cc:1397] T 589afb72b1a44f1dba4cad902f5a12f6 P ea9430b6ad3f4203bf91e28f032217eb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 17:06:30.016302 27772 tablet_bootstrap.cc:654] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:30.017891 27772 log.cc:824] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:30.018996 27770 raft_consensus.cc:348] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9430b6ad3f4203bf91e28f032217eb" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42463 } } peers { permanent_uuid: "a5f6744d1986420f8ef2ecdab790fa7b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33229 } } peers { permanent_uuid: "83c1d02f99674814866afaa7e2c14622" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37845 } }
I20260806 17:06:30.019322 27770 raft_consensus.cc:374] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:30.019404 27770 raft_consensus.cc:729] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f6744d1986420f8ef2ecdab790fa7b, State: Initialized, Role: FOLLOWER
I20260806 17:06:30.020030 27770 consensus_queue.cc:260] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9430b6ad3f4203bf91e28f032217eb" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42463 } } peers { permanent_uuid: "a5f6744d1986420f8ef2ecdab790fa7b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33229 } } peers { permanent_uuid: "83c1d02f99674814866afaa7e2c14622" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37845 } }
I20260806 17:06:30.020846 27772 tablet_bootstrap.cc:492] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622: No bootstrap required, opened a new log
I20260806 17:06:30.020808 27771 raft_consensus.cc:348] T 589afb72b1a44f1dba4cad902f5a12f6 P ea9430b6ad3f4203bf91e28f032217eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9430b6ad3f4203bf91e28f032217eb" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42463 } } peers { permanent_uuid: "a5f6744d1986420f8ef2ecdab790fa7b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33229 } } peers { permanent_uuid: "83c1d02f99674814866afaa7e2c14622" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37845 } }
I20260806 17:06:30.021090 27772 ts_tablet_manager.cc:1397] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 17:06:30.021130 27771 raft_consensus.cc:374] T 589afb72b1a44f1dba4cad902f5a12f6 P ea9430b6ad3f4203bf91e28f032217eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:30.021200 27771 raft_consensus.cc:729] T 589afb72b1a44f1dba4cad902f5a12f6 P ea9430b6ad3f4203bf91e28f032217eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9430b6ad3f4203bf91e28f032217eb, State: Initialized, Role: FOLLOWER
I20260806 17:06:30.021399 27770 ts_tablet_manager.cc:1428] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 17:06:30.021772 27771 consensus_queue.cc:260] T 589afb72b1a44f1dba4cad902f5a12f6 P ea9430b6ad3f4203bf91e28f032217eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9430b6ad3f4203bf91e28f032217eb" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42463 } } peers { permanent_uuid: "a5f6744d1986420f8ef2ecdab790fa7b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33229 } } peers { permanent_uuid: "83c1d02f99674814866afaa7e2c14622" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37845 } }
I20260806 17:06:30.021960 27616 heartbeater.cc:503] Master 127.9.111.126:38169 was elected leader, sending a full tablet report...
I20260806 17:06:30.022567 27479 heartbeater.cc:503] Master 127.9.111.126:38169 was elected leader, sending a full tablet report...
I20260806 17:06:30.023048 27771 ts_tablet_manager.cc:1428] T 589afb72b1a44f1dba4cad902f5a12f6 P ea9430b6ad3f4203bf91e28f032217eb: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 17:06:30.027048 27772 raft_consensus.cc:348] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9430b6ad3f4203bf91e28f032217eb" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42463 } } peers { permanent_uuid: "a5f6744d1986420f8ef2ecdab790fa7b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33229 } } peers { permanent_uuid: "83c1d02f99674814866afaa7e2c14622" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37845 } }
I20260806 17:06:30.027535 27772 raft_consensus.cc:374] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:30.027633 27772 raft_consensus.cc:729] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c1d02f99674814866afaa7e2c14622, State: Initialized, Role: FOLLOWER
I20260806 17:06:30.028169 27772 consensus_queue.cc:260] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9430b6ad3f4203bf91e28f032217eb" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42463 } } peers { permanent_uuid: "a5f6744d1986420f8ef2ecdab790fa7b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33229 } } peers { permanent_uuid: "83c1d02f99674814866afaa7e2c14622" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37845 } }
I20260806 17:06:30.029002 27753 heartbeater.cc:503] Master 127.9.111.126:38169 was elected leader, sending a full tablet report...
I20260806 17:06:30.029388 27772 ts_tablet_manager.cc:1428] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260806 17:06:30.071451 27778 raft_consensus.cc:482] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:06:30.071658 27778 raft_consensus.cc:504] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9430b6ad3f4203bf91e28f032217eb" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42463 } } peers { permanent_uuid: "a5f6744d1986420f8ef2ecdab790fa7b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33229 } } peers { permanent_uuid: "83c1d02f99674814866afaa7e2c14622" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37845 } }
I20260806 17:06:30.072767 27778 leader_election.cc:288] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea9430b6ad3f4203bf91e28f032217eb (127.9.111.65:42463), a5f6744d1986420f8ef2ecdab790fa7b (127.9.111.66:33229)
I20260806 17:06:30.077242 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:30.073195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:54443 (local address 127.9.111.66:33229)
0806 17:06:30.073324 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:30.073327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:30.073340 (+ 13us) server_negotiation.cc:407] Connection header received
0806 17:06:30.073389 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:30.073391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:30.073431 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:30.073506 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:30.073886 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.074590 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.076195 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.076611 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.076753 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:30.076819 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:30.077025 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 17:06:30.077102 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 17:06:30.077304 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:30.073039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:42463 (local address 127.9.111.67:49195)
0806 17:06:30.073754 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:30.073795 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:30.073932 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:30.074169 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:30.074172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:30.074189 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:30.074319 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:30.074325 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.075086 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.075090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:30.076119 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:30.076129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.076830 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.076834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:30.076942 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:30.076963 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 17:06:30.077061 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 17:06:30.077133 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":294,"thread_start_us":170,"threads_started":1}
I20260806 17:06:30.077287 27765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:30.073175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:49195 (local address 127.9.111.65:42463)
0806 17:06:30.073321 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:30.073326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:30.073885 (+ 559us) server_negotiation.cc:407] Connection header received
0806 17:06:30.074029 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:30.074031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:30.074071 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:30.074130 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:30.074424 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.074969 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.076261 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.076703 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.076837 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:30.076893 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:30.077077 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 17:06:30.077157 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 17:06:30.077991 27432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589afb72b1a44f1dba4cad902f5a12f6" candidate_uuid: "83c1d02f99674814866afaa7e2c14622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea9430b6ad3f4203bf91e28f032217eb" is_pre_election: true
I20260806 17:06:30.078171 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:30.073039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:33229 (local address 127.9.111.67:54443)
0806 17:06:30.073151 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:30.073167 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:30.073262 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:30.073550 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:30.073554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:30.073577 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:30.073710 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:30.073718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.074700 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.074705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:30.076065 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:30.076076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.076753 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.076757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:30.076886 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:30.076916 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 17:06:30.077007 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 17:06:30.077087 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260806 17:06:30.078364 27432 raft_consensus.cc:2454] T 589afb72b1a44f1dba4cad902f5a12f6 P ea9430b6ad3f4203bf91e28f032217eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c1d02f99674814866afaa7e2c14622 in term 0.
I20260806 17:06:30.078897 27639 leader_election.cc:302] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83c1d02f99674814866afaa7e2c14622, ea9430b6ad3f4203bf91e28f032217eb; no voters:
I20260806 17:06:30.078960 27568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589afb72b1a44f1dba4cad902f5a12f6" candidate_uuid: "83c1d02f99674814866afaa7e2c14622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f6744d1986420f8ef2ecdab790fa7b" is_pre_election: true
I20260806 17:06:30.079183 27778 raft_consensus.cc:2781] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:06:30.079286 27778 raft_consensus.cc:482] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:06:30.079247 27568 raft_consensus.cc:2454] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c1d02f99674814866afaa7e2c14622 in term 0.
I20260806 17:06:30.079321 27778 raft_consensus.cc:3037] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:30.080577 27778 raft_consensus.cc:504] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9430b6ad3f4203bf91e28f032217eb" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42463 } } peers { permanent_uuid: "a5f6744d1986420f8ef2ecdab790fa7b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33229 } } peers { permanent_uuid: "83c1d02f99674814866afaa7e2c14622" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37845 } }
I20260806 17:06:30.081004 27778 leader_election.cc:288] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [CANDIDATE]: Term 1 election: Requested vote from peers ea9430b6ad3f4203bf91e28f032217eb (127.9.111.65:42463), a5f6744d1986420f8ef2ecdab790fa7b (127.9.111.66:33229)
I20260806 17:06:30.081390 27432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589afb72b1a44f1dba4cad902f5a12f6" candidate_uuid: "83c1d02f99674814866afaa7e2c14622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea9430b6ad3f4203bf91e28f032217eb"
I20260806 17:06:30.081439 27568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589afb72b1a44f1dba4cad902f5a12f6" candidate_uuid: "83c1d02f99674814866afaa7e2c14622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f6744d1986420f8ef2ecdab790fa7b"
I20260806 17:06:30.081554 27568 raft_consensus.cc:3037] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:30.081553 27432 raft_consensus.cc:3037] T 589afb72b1a44f1dba4cad902f5a12f6 P ea9430b6ad3f4203bf91e28f032217eb [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:30.083029 27568 raft_consensus.cc:2454] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c1d02f99674814866afaa7e2c14622 in term 1.
I20260806 17:06:30.083029 27432 raft_consensus.cc:2454] T 589afb72b1a44f1dba4cad902f5a12f6 P ea9430b6ad3f4203bf91e28f032217eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c1d02f99674814866afaa7e2c14622 in term 1.
I20260806 17:06:30.083489 27637 leader_election.cc:302] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83c1d02f99674814866afaa7e2c14622, a5f6744d1986420f8ef2ecdab790fa7b; no voters:
I20260806 17:06:30.083745 27778 raft_consensus.cc:2781] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:30.083956 27778 raft_consensus.cc:686] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [term 1 LEADER]: Becoming Leader. State: Replica: 83c1d02f99674814866afaa7e2c14622, State: Running, Role: LEADER
I20260806 17:06:30.084324 27778 consensus_queue.cc:237] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9430b6ad3f4203bf91e28f032217eb" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42463 } } peers { permanent_uuid: "a5f6744d1986420f8ef2ecdab790fa7b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33229 } } peers { permanent_uuid: "83c1d02f99674814866afaa7e2c14622" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37845 } }
I20260806 17:06:30.087735 27285 catalog_manager.cc:5495] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c1d02f99674814866afaa7e2c14622 (127.9.111.67). New cstate: current_term: 1 leader_uuid: "83c1d02f99674814866afaa7e2c14622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9430b6ad3f4203bf91e28f032217eb" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f6744d1986420f8ef2ecdab790fa7b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c1d02f99674814866afaa7e2c14622" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37845 } health_report { overall_health: HEALTHY } } }
I20260806 17:06:30.123320 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:30.119614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41412 (local address 127.9.111.67:37845)
0806 17:06:30.119756 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:30.119760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:30.119774 (+ 14us) server_negotiation.cc:407] Connection header received
0806 17:06:30.119905 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:30.119907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:30.119963 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:30.120053 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:06:30.120320 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.120820 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.121745 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.121997 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.122078 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:30.122391 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:06:30.122826 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:06:30.122919 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:30.123026 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 17:06:30.123081 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 17:06:30.129658 27432 raft_consensus.cc:1261] T 589afb72b1a44f1dba4cad902f5a12f6 P ea9430b6ad3f4203bf91e28f032217eb [term 1 FOLLOWER]: Refusing update from remote peer 83c1d02f99674814866afaa7e2c14622: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:06:30.129658 27568 raft_consensus.cc:1261] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b [term 1 FOLLOWER]: Refusing update from remote peer 83c1d02f99674814866afaa7e2c14622: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:06:30.130439 27778 consensus_queue.cc:1035] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f6744d1986420f8ef2ecdab790fa7b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:06:30.130666 27780 consensus_queue.cc:1035] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea9430b6ad3f4203bf91e28f032217eb" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:06:30.137957 27783 mvcc.cc:205] Tried to move back new op lower bound from 7315603415562874880 to 7315603415388610560. Current Snapshot: MvccSnapshot[applied={T|T < 7315603415562874880}]
I20260806 17:06:30.139597 27784 mvcc.cc:205] Tried to move back new op lower bound from 7315603415562874880 to 7315603415388610560. Current Snapshot: MvccSnapshot[applied={T|T < 7315603415562874880}]
I20260806 17:06:30.148145 27285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:06:30.148361 27285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:06:30.151175 27285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9dce969061c5410cbbe7ebdf75693321: Sending DeleteTablet for 3 replicas of tablet 589afb72b1a44f1dba4cad902f5a12f6
I20260806 17:06:30.152125 27548 tablet_service.cc:1514] Processing DeleteTablet for tablet 589afb72b1a44f1dba4cad902f5a12f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:06:30 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:53290
I20260806 17:06:30.152096 27412 tablet_service.cc:1514] Processing DeleteTablet for tablet 589afb72b1a44f1dba4cad902f5a12f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:06:30 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:40840
I20260806 17:06:30.152480 27685 tablet_service.cc:1514] Processing DeleteTablet for tablet 589afb72b1a44f1dba4cad902f5a12f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:06:30 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:41408
I20260806 17:06:30.152776 27793 tablet_replica.cc:330] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b: stopping tablet replica
I20260806 17:06:30.153138 27795 tablet_replica.cc:330] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622: stopping tablet replica
I20260806 17:06:30.153179 27793 raft_consensus.cc:2229] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:06:30.153316 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 27348
I20260806 17:06:30.153585 27795 raft_consensus.cc:2229] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:06:30.153586 27793 raft_consensus.cc:2258] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:06:30.154759 27795 raft_consensus.cc:2258] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:06:30.155663 27793 ts_tablet_manager.cc:1905] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:06:30.158553 27795 ts_tablet_manager.cc:1905] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:06:30.159057 27793 ts_tablet_manager.cc:1918] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:06:30.159154 27793 log.cc:1196] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/589afb72b1a44f1dba4cad902f5a12f6
I20260806 17:06:30.159473 27793 ts_tablet_manager.cc:1939] T 589afb72b1a44f1dba4cad902f5a12f6 P a5f6744d1986420f8ef2ecdab790fa7b: Deleting consensus metadata
I20260806 17:06:30.161038 27269 catalog_manager.cc:4841] TS a5f6744d1986420f8ef2ecdab790fa7b (127.9.111.66:33229): tablet 589afb72b1a44f1dba4cad902f5a12f6 (table test-table [id=f333396cc6c640819bad15afb1581be0]) successfully deleted
I20260806 17:06:30.161633 27795 ts_tablet_manager.cc:1918] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:06:30.161840 27795 log.cc:1196] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/589afb72b1a44f1dba4cad902f5a12f6
I20260806 17:06:30.162236 27795 ts_tablet_manager.cc:1939] T 589afb72b1a44f1dba4cad902f5a12f6 P 83c1d02f99674814866afaa7e2c14622: Deleting consensus metadata
W20260806 17:06:30.163332 27271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.111.65:42463 (error 108)
I20260806 17:06:30.163801 27272 catalog_manager.cc:4841] TS 83c1d02f99674814866afaa7e2c14622 (127.9.111.67:37845): tablet 589afb72b1a44f1dba4cad902f5a12f6 (table test-table [id=f333396cc6c640819bad15afb1581be0]) successfully deleted
I20260806 17:06:30.164300 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 27484
I20260806 17:06:30.165988 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:30.164627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:42463 (local address 127.0.0.1:40850)
0806 17:06:30.165740 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:30.165869 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.65:42463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1008}
W20260806 17:06:30.168984 27271 catalog_manager.cc:4584] TS ea9430b6ad3f4203bf91e28f032217eb (127.9.111.65:42463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 589afb72b1a44f1dba4cad902f5a12f6: Network error: Client connection negotiation failed: client connection to 127.9.111.65:42463: connect: Connection refused (error 111)
I20260806 17:06:30.171759 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 27621
I20260806 17:06:30.178191 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 27253
2026-08-06T17:06:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2412 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T17:06:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:30Z Disabled control of system clock
I20260806 17:06:30.211438 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:38721
--webserver_interface=127.9.111.126
--webserver_port=0
--builtin_ntp_servers=127.9.111.84:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:38721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 17:06:30.325667 27800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:30.326015 27800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:30.326054 27800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:30.330446 27800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:30.330533 27800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:30.330549 27800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:30.330566 27800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:30.330581 27800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:30.335786 27800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:38721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:38721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:30.337126 27800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:30.338683 27800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:30.347922 27806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:30.347922 27808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:30.347962 27805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:30.348304 27800 server_base.cc:1034] running on GCE node
I20260806 17:06:30.349002 27800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:30.350296 27800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:30.351482 27800 hybrid_clock.cc:648] HybridClock initialized: now 1786035990351456 us; error 40 us; skew 500 ppm
I20260806 17:06:30.353694 27800 webserver.cc:460] Webserver started at http://127.9.111.126:37447/ using document root <none> and password file <none>
I20260806 17:06:30.354375 27800 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:30.354453 27800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:30.354638 27800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:30.356835 27800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "34f90341a28842438950bb7602804531"
format_stamp: "Formatted at 2026-08-06 17:06:30 on dist-test-slave-69rg"
server_key: "5c01b65efcad8c33078c00d81d2a703f"
server_key_iv: "25ca8ab34721f6649d36a70b886b15c7"
server_key_version: "encryptionkey@0"
I20260806 17:06:30.357498 27800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "34f90341a28842438950bb7602804531"
format_stamp: "Formatted at 2026-08-06 17:06:30 on dist-test-slave-69rg"
server_key: "5c01b65efcad8c33078c00d81d2a703f"
server_key_iv: "25ca8ab34721f6649d36a70b886b15c7"
server_key_version: "encryptionkey@0"
I20260806 17:06:30.361429 27800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 17:06:30.364033 27814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:30.365276 27800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 17:06:30.365425 27800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "34f90341a28842438950bb7602804531"
format_stamp: "Formatted at 2026-08-06 17:06:30 on dist-test-slave-69rg"
server_key: "5c01b65efcad8c33078c00d81d2a703f"
server_key_iv: "25ca8ab34721f6649d36a70b886b15c7"
server_key_version: "encryptionkey@0"
I20260806 17:06:30.365545 27800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:30.392573 27800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:30.393455 27800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:30.393687 27800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:30.401399 27800 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:38721
I20260806 17:06:30.401410 27868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:38721 every 8 connection(s)
I20260806 17:06:30.402475 27800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:30.405695 27869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:30.408452 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 27800
I20260806 17:06:30.408609 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:30.408938 9661 external_mini_cluster.cc:1428] Setting key 762b9c74d687a6192da62af237005a15
I20260806 17:06:30.412528 27869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531: Bootstrap starting.
I20260806 17:06:30.414925 27869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:30.415712 27869 log.cc:824] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:30.417887 27869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531: No bootstrap required, opened a new log
I20260806 17:06:30.418988 27872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:30.410865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47734 (local address 127.9.111.126:38721)
0806 17:06:30.411512 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:30.411527 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:30.411564 (+ 37us) server_negotiation.cc:407] Connection header received
0806 17:06:30.412181 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:30.412197 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:30.412585 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:30.412992 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:30.413445 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.414514 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.415548 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.416091 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.416274 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:30.417170 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:30.417175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:30.417189 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:30.417587 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:30.417642 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:30.417663 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:30.417785 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:30.417967 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 17:06:30.418283 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":141,"threads_started":1}
I20260806 17:06:30.422044 27869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f90341a28842438950bb7602804531" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38721 } }
I20260806 17:06:30.422307 27869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:30.422389 27869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f90341a28842438950bb7602804531, State: Initialized, Role: FOLLOWER
I20260806 17:06:30.422864 27869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f90341a28842438950bb7602804531" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38721 } }
I20260806 17:06:30.423019 27869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:30.423110 27869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:30.423213 27869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:30.424188 27869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f90341a28842438950bb7602804531" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38721 } }
I20260806 17:06:30.424536 27869 leader_election.cc:302] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34f90341a28842438950bb7602804531; no voters:
I20260806 17:06:30.424882 27869 leader_election.cc:288] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:30.425040 27874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:30.425320 27874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531 [term 1 LEADER]: Becoming Leader. State: Replica: 34f90341a28842438950bb7602804531, State: Running, Role: LEADER
I20260806 17:06:30.425637 27874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f90341a28842438950bb7602804531" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38721 } }
I20260806 17:06:30.426221 27869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:30.427241 27875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34f90341a28842438950bb7602804531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f90341a28842438950bb7602804531" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38721 } } }
I20260806 17:06:30.427382 27875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:30.427758 27876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34f90341a28842438950bb7602804531. Latest consensus state: current_term: 1 leader_uuid: "34f90341a28842438950bb7602804531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f90341a28842438950bb7602804531" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 38721 } } }
I20260806 17:06:30.427870 27876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:30.428179 27883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:30.431173 27883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:30.436432 27883 catalog_manager.cc:1347] Generated new cluster ID: 94626b6a57104598add785a67157af06
I20260806 17:06:30.436506 27883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:30.450388 27883 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:30.451320 27883 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:30.456220 27883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531: Generated new TSK 0
I20260806 17:06:30.457162 27883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:06:30.465695 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38721
--builtin_ntp_servers=127.9.111.84:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 17:06:30.588143 27893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:30.588434 27893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:30.588475 27893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:30.592723 27893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:30.592810 27893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:30.592877 27893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:06:30.598124 27893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:30.599406 27893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:30.600883 27893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 17:06:30.613029 27893 server_base.cc:1034] running on GCE node
W20260806 17:06:30.612847 27899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:30.612847 27898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:30.612972 27901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:30.613979 27893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:30.614928 27893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:30.616411 27893 hybrid_clock.cc:648] HybridClock initialized: now 1786035990616342 us; error 89 us; skew 500 ppm
I20260806 17:06:30.619580 27893 webserver.cc:460] Webserver started at http://127.9.111.65:42303/ using document root <none> and password file <none>
I20260806 17:06:30.620675 27893 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:30.620774 27893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:30.620990 27893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:30.624603 27893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "956b8702e0334a36b30d3172e4515a73"
format_stamp: "Formatted at 2026-08-06 17:06:30 on dist-test-slave-69rg"
server_key: "d6284779b5bcae0baf86edcb38621bbf"
server_key_iv: "83c22ec6c40b5e23f735716d1ec63ad7"
server_key_version: "encryptionkey@0"
I20260806 17:06:30.625499 27893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "956b8702e0334a36b30d3172e4515a73"
format_stamp: "Formatted at 2026-08-06 17:06:30 on dist-test-slave-69rg"
server_key: "d6284779b5bcae0baf86edcb38621bbf"
server_key_iv: "83c22ec6c40b5e23f735716d1ec63ad7"
server_key_version: "encryptionkey@0"
I20260806 17:06:30.631798 27893 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 17:06:30.635200 27907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:30.636893 27893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 17:06:30.637107 27893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "956b8702e0334a36b30d3172e4515a73"
format_stamp: "Formatted at 2026-08-06 17:06:30 on dist-test-slave-69rg"
server_key: "d6284779b5bcae0baf86edcb38621bbf"
server_key_iv: "83c22ec6c40b5e23f735716d1ec63ad7"
server_key_version: "encryptionkey@0"
I20260806 17:06:30.637287 27893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:30.666807 27893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:30.667732 27893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:30.667985 27893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:30.668748 27893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:30.670084 27893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:30.670177 27893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:30.670251 27893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:30.670308 27893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:30.680061 27893 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:42359
I20260806 17:06:30.680428 28022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:42359 every 8 connection(s)
I20260806 17:06:30.681089 27893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:06:30.685182 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 27893
I20260806 17:06:30.685335 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:06:30.685629 9661 external_mini_cluster.cc:1428] Setting key fc026d539f96842185acc7e112483195
I20260806 17:06:30.688083 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38721
--builtin_ntp_servers=127.9.111.84:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 17:06:30.692071 27872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:30.683120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:58351 (local address 127.9.111.126:38721)
0806 17:06:30.683308 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:30.683312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:30.684331 (+ 1019us) server_negotiation.cc:407] Connection header received
0806 17:06:30.686012 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:30.686015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:30.686084 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:30.686182 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:30.687495 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.688306 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.689562 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.689964 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.690051 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:30.691571 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:30.691574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:30.691576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:30.691589 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:30.691610 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:30.691617 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:30.691678 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:30.691895 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 17:06:30.691951 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 17:06:30.693579 28025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:30.683446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38721 (local address 127.9.111.65:58351)
0806 17:06:30.684114 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:30.684158 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:30.685673 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:30.686449 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:30.686457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:30.686569 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:30.687314 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:30.687339 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.688457 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.688460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:30.689423 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:30.689434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.690159 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.690168 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:30.690437 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:30.691204 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:06:30.691208 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:30.691283 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:06:30.691386 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:30.691714 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:30.691728 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:30.691737 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 17:06:30.691890 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 17:06:30.692237 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":119,"threads_started":1}
I20260806 17:06:30.695128 28023 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38721
I20260806 17:06:30.695415 28023 heartbeater.cc:464] Registering TS with master...
I20260806 17:06:30.695973 28023 heartbeater.cc:511] Master 127.9.111.126:38721 requested a full tablet report, sending...
I20260806 17:06:30.697628 27832 ts_manager.cc:194] Registered new tserver with Master: 956b8702e0334a36b30d3172e4515a73 (127.9.111.65:42359)
I20260806 17:06:30.698900 27832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.111.65:58351
W20260806 17:06:30.821626 28026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:30.822018 28026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:30.822891 28026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:30.831524 28026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:30.831935 28026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:30.832116 28026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:06:30.838276 28026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:30.839577 28026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:30.841150 28026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:30.853006 28031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:30.853266 28032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:30.853328 28034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:30.853533 28026 server_base.cc:1034] running on GCE node
I20260806 17:06:30.854127 28026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:30.854957 28026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:30.856190 28026 hybrid_clock.cc:648] HybridClock initialized: now 1786035990856167 us; error 26 us; skew 500 ppm
I20260806 17:06:30.858752 28026 webserver.cc:460] Webserver started at http://127.9.111.66:36277/ using document root <none> and password file <none>
I20260806 17:06:30.859472 28026 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:30.859588 28026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:30.859874 28026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:30.862617 28026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "1879fc24146948cebd593f0c0d195642"
format_stamp: "Formatted at 2026-08-06 17:06:30 on dist-test-slave-69rg"
server_key: "1b9b141a02c0758a2c29adc156ffb382"
server_key_iv: "e411fbe614095e827ba25abf43c5323e"
server_key_version: "encryptionkey@0"
I20260806 17:06:30.863337 28026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "1879fc24146948cebd593f0c0d195642"
format_stamp: "Formatted at 2026-08-06 17:06:30 on dist-test-slave-69rg"
server_key: "1b9b141a02c0758a2c29adc156ffb382"
server_key_iv: "e411fbe614095e827ba25abf43c5323e"
server_key_version: "encryptionkey@0"
I20260806 17:06:30.867821 28026 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 17:06:30.870996 28040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:30.873055 28026 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:06:30.873332 28026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "1879fc24146948cebd593f0c0d195642"
format_stamp: "Formatted at 2026-08-06 17:06:30 on dist-test-slave-69rg"
server_key: "1b9b141a02c0758a2c29adc156ffb382"
server_key_iv: "e411fbe614095e827ba25abf43c5323e"
server_key_version: "encryptionkey@0"
I20260806 17:06:30.873535 28026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:30.902635 28026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:30.903520 28026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:30.903752 28026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:30.904424 28026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:30.905476 28026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:30.905553 28026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:30.905623 28026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:30.905670 28026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:30.919688 28026 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:34997
I20260806 17:06:30.919821 28155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:34997 every 8 connection(s)
I20260806 17:06:30.921212 28026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:06:30.928995 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 28026
I20260806 17:06:30.929165 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:06:30.929669 9661 external_mini_cluster.cc:1428] Setting key 31b13e3028ea5fa0060387eb7cd599a8
I20260806 17:06:30.932691 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38721
--builtin_ntp_servers=127.9.111.84:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 17:06:30.933735 27872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:30.923957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:47013 (local address 127.9.111.126:38721)
0806 17:06:30.924146 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:30.924152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:30.925336 (+ 1184us) server_negotiation.cc:407] Connection header received
0806 17:06:30.926981 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:30.926985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:30.927074 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:30.927210 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:30.928451 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.929137 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.930873 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.931136 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.931276 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:30.932791 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:30.932795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:30.932803 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:30.932818 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:30.932857 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:30.932868 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:30.932936 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:30.933303 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 17:06:30.933435 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 17:06:30.934835 28158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:30.924376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38721 (local address 127.9.111.66:47013)
0806 17:06:30.925162 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:30.925208 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:30.926697 (+ 1489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:30.927399 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:30.927406 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:30.927601 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:30.928286 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:30.928302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.929353 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.929358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:30.930714 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:30.930725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:30.931328 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:30.931332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:30.931544 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:30.932486 (+ 942us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:06:30.932488 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:30.932535 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:06:30.932611 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:30.933008 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:30.933021 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:30.933031 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 17:06:30.933189 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 17:06:30.933555 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":151,"threads_started":1}
I20260806 17:06:30.936542 28156 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38721
I20260806 17:06:30.936806 28156 heartbeater.cc:464] Registering TS with master...
I20260806 17:06:30.937479 28156 heartbeater.cc:511] Master 127.9.111.126:38721 requested a full tablet report, sending...
I20260806 17:06:30.939097 27832 ts_manager.cc:194] Registered new tserver with Master: 1879fc24146948cebd593f0c0d195642 (127.9.111.66:34997)
I20260806 17:06:30.939796 27832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.111.66:47013
W20260806 17:06:31.067481 28159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:31.067850 28159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:31.067929 28159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:31.072484 28159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:31.072577 28159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:31.072667 28159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:06:31.079263 28159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:38721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:31.081544 28159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:31.083222 28159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:31.095064 28164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:31.095088 28167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:31.095085 28165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:31.095589 28159 server_base.cc:1034] running on GCE node
I20260806 17:06:31.096292 28159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:31.097317 28159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:31.098670 28159 hybrid_clock.cc:648] HybridClock initialized: now 1786035991098651 us; error 36 us; skew 500 ppm
I20260806 17:06:31.100921 28159 webserver.cc:460] Webserver started at http://127.9.111.67:35305/ using document root <none> and password file <none>
I20260806 17:06:31.102042 28159 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:31.102159 28159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:31.102425 28159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:31.104732 28159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "d2e31af9fbd34917977aae518bfd57a5"
format_stamp: "Formatted at 2026-08-06 17:06:31 on dist-test-slave-69rg"
server_key: "87827de02718e8fdab9a3232f2f6d472"
server_key_iv: "88dd2117ec22465e84627aa636135969"
server_key_version: "encryptionkey@0"
I20260806 17:06:31.105376 28159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "d2e31af9fbd34917977aae518bfd57a5"
format_stamp: "Formatted at 2026-08-06 17:06:31 on dist-test-slave-69rg"
server_key: "87827de02718e8fdab9a3232f2f6d472"
server_key_iv: "88dd2117ec22465e84627aa636135969"
server_key_version: "encryptionkey@0"
I20260806 17:06:31.109544 28159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 17:06:31.112495 28173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:31.114629 28159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:06:31.114948 28159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "d2e31af9fbd34917977aae518bfd57a5"
format_stamp: "Formatted at 2026-08-06 17:06:31 on dist-test-slave-69rg"
server_key: "87827de02718e8fdab9a3232f2f6d472"
server_key_iv: "88dd2117ec22465e84627aa636135969"
server_key_version: "encryptionkey@0"
I20260806 17:06:31.115218 28159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:31.127426 28159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:31.128193 28159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:31.128448 28159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:31.129063 28159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:31.130102 28159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:31.130178 28159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:31.130252 28159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:31.130306 28159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:31.139891 28159 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:37049
I20260806 17:06:31.139914 28288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:37049 every 8 connection(s)
I20260806 17:06:31.140882 28159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:06:31.143860 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 28159
I20260806 17:06:31.144004 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:06:31.144369 9661 external_mini_cluster.cc:1428] Setting key ada857ca0d32c2d781b01818d8dcfe58
I20260806 17:06:31.152462 27872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:31.143289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:53487 (local address 127.9.111.126:38721)
0806 17:06:31.143468 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:31.143474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:31.145225 (+ 1751us) server_negotiation.cc:407] Connection header received
0806 17:06:31.147145 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:31.147148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:31.147204 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:31.147272 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:31.148285 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.148897 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.150303 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.150533 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.150625 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:31.151891 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:31.151895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:31.151897 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:31.151911 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:31.151930 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:31.151938 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:31.152011 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:31.152220 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 17:06:31.152313 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 17:06:31.153668 28291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:31.143986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:38721 (local address 127.9.111.67:53487)
0806 17:06:31.144983 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:31.145065 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:31.146956 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:31.147453 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:31.147460 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:31.147551 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:31.148083 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:31.148138 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.149040 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.149042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:31.150113 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:31.150132 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.150695 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.150699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:31.150850 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:31.151591 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:06:31.151593 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:31.151640 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:06:31.151714 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:31.152058 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:31.152067 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:31.152072 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 17:06:31.152199 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 17:06:31.152607 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":169,"threads_started":1}
I20260806 17:06:31.155215 28289 heartbeater.cc:347] Connected to a master server at 127.9.111.126:38721
I20260806 17:06:31.155536 28289 heartbeater.cc:464] Registering TS with master...
I20260806 17:06:31.156139 28289 heartbeater.cc:511] Master 127.9.111.126:38721 requested a full tablet report, sending...
I20260806 17:06:31.157342 27832 ts_manager.cc:194] Registered new tserver with Master: d2e31af9fbd34917977aae518bfd57a5 (127.9.111.67:37049)
I20260806 17:06:31.158155 27832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.111.67:53487
I20260806 17:06:31.159209 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 17:06:31.165480 27872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:31.161388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47736 (local address 127.9.111.126:38721)
0806 17:06:31.161530 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:31.161534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:31.161718 (+ 184us) server_negotiation.cc:407] Connection header received
0806 17:06:31.161804 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:31.161806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:31.161865 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:31.161932 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:31.162189 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.162685 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.163678 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.163910 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.164003 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:31.164943 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:31.164946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:31.164948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:31.164961 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:31.164980 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:31.164988 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:31.165051 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:31.165133 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 17:06:31.165199 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 17:06:31.168936 27832 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:06:31.172264 27832 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:06:31.187846 28305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:31.183068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52808 (local address 127.9.111.66:34997)
0806 17:06:31.183368 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:31.183381 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:31.183685 (+ 304us) server_negotiation.cc:407] Connection header received
0806 17:06:31.183764 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:31.183769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:31.183938 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:31.184082 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:31.184333 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.185378 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.186546 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.187095 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.187228 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:31.187451 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:31.187529 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 17:06:31.187651 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":123,"threads_started":1}
I20260806 17:06:31.187865 28304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:31.182440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:37049 (local address 127.0.0.1:46888)
0806 17:06:31.182994 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:31.183008 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:31.183103 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:31.183607 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:31.183610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:31.183628 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:31.183710 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:31.183718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.185092 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.185095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:31.186185 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:31.186195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.187278 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.187281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:31.187414 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:31.187440 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 17:06:31.187488 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 17:06:31.187690 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":50,"thread_start_us":101,"threads_started":1}
I20260806 17:06:31.187863 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:31.182773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:34997 (local address 127.0.0.1:52808)
0806 17:06:31.183468 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:31.183480 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:31.183592 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:31.184093 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:31.184096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:31.184116 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:31.184197 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:31.184202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.185497 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.185503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:31.186410 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:31.186420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.187232 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.187235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:31.187343 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:31.187381 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 17:06:31.187501 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 17:06:31.187694 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":50,"thread_start_us":93,"threads_started":1}
I20260806 17:06:31.188444 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:31.182593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46888 (local address 127.9.111.67:37049)
0806 17:06:31.183156 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:31.183167 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:31.183189 (+ 22us) server_negotiation.cc:407] Connection header received
0806 17:06:31.183274 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:31.183278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:31.183448 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:31.183628 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:31.183844 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.184788 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.186324 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.187126 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.187929 (+ 803us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:31.188105 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:31.188156 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 17:06:31.188265 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":115,"threads_started":1}
I20260806 17:06:31.189010 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:31.181705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:42359 (local address 127.0.0.1:45692)
0806 17:06:31.182233 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:31.182292 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:31.182422 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:31.183265 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:31.183271 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:31.183295 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:31.183433 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:31.183440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.184925 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.184928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:31.187299 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:31.187307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.188612 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.188616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:31.188706 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:31.188727 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 17:06:31.188819 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 17:06:31.188886 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":39,"thread_start_us":79,"threads_started":1}
I20260806 17:06:31.189144 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:31.181882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.9.111.65:42359)
0806 17:06:31.182188 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:31.182196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:31.182572 (+ 376us) server_negotiation.cc:407] Connection header received
0806 17:06:31.182661 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:31.182666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:31.182921 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:31.183082 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:31.183572 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.184756 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.187441 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.188258 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.188371 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:31.188541 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:31.188866 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 17:06:31.188980 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":100,"threads_started":1}
I20260806 17:06:31.191356 28222 tablet_service.cc:1467] Processing CreateTablet for tablet c4221afb9e464063994db8737c8e67c9 (DEFAULT_TABLE table=test-table [id=93b238d27b2945ceaa436c6d5b220239]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:06:31.191357 28090 tablet_service.cc:1467] Processing CreateTablet for tablet c4221afb9e464063994db8737c8e67c9 (DEFAULT_TABLE table=test-table [id=93b238d27b2945ceaa436c6d5b220239]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:06:31.192376 28090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4221afb9e464063994db8737c8e67c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:31.192394 28222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4221afb9e464063994db8737c8e67c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:31.194790 27956 tablet_service.cc:1467] Processing CreateTablet for tablet c4221afb9e464063994db8737c8e67c9 (DEFAULT_TABLE table=test-table [id=93b238d27b2945ceaa436c6d5b220239]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:06:31.195922 27956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4221afb9e464063994db8737c8e67c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:31.199141 28306 tablet_bootstrap.cc:492] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5: Bootstrap starting.
I20260806 17:06:31.200349 28307 tablet_bootstrap.cc:492] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642: Bootstrap starting.
I20260806 17:06:31.201373 28306 tablet_bootstrap.cc:654] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:31.202203 28306 log.cc:824] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:31.202466 28307 tablet_bootstrap.cc:654] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:31.203251 28307 log.cc:824] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:31.203267 28309 tablet_bootstrap.cc:492] T c4221afb9e464063994db8737c8e67c9 P 956b8702e0334a36b30d3172e4515a73: Bootstrap starting.
I20260806 17:06:31.204315 28306 tablet_bootstrap.cc:492] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5: No bootstrap required, opened a new log
I20260806 17:06:31.204532 28306 ts_tablet_manager.cc:1397] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 17:06:31.205619 28307 tablet_bootstrap.cc:492] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642: No bootstrap required, opened a new log
I20260806 17:06:31.205839 28307 ts_tablet_manager.cc:1397] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 17:06:31.205909 28309 tablet_bootstrap.cc:654] T c4221afb9e464063994db8737c8e67c9 P 956b8702e0334a36b30d3172e4515a73: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:31.206719 28309 log.cc:824] T c4221afb9e464063994db8737c8e67c9 P 956b8702e0334a36b30d3172e4515a73: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:31.208765 28306 raft_consensus.cc:348] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956b8702e0334a36b30d3172e4515a73" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42359 } } peers { permanent_uuid: "d2e31af9fbd34917977aae518bfd57a5" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37049 } } peers { permanent_uuid: "1879fc24146948cebd593f0c0d195642" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34997 } }
I20260806 17:06:31.209067 28309 tablet_bootstrap.cc:492] T c4221afb9e464063994db8737c8e67c9 P 956b8702e0334a36b30d3172e4515a73: No bootstrap required, opened a new log
I20260806 17:06:31.209116 28306 raft_consensus.cc:374] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:31.209182 28306 raft_consensus.cc:729] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2e31af9fbd34917977aae518bfd57a5, State: Initialized, Role: FOLLOWER
I20260806 17:06:31.209261 28309 ts_tablet_manager.cc:1397] T c4221afb9e464063994db8737c8e67c9 P 956b8702e0334a36b30d3172e4515a73: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260806 17:06:31.209314 28307 raft_consensus.cc:348] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956b8702e0334a36b30d3172e4515a73" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42359 } } peers { permanent_uuid: "d2e31af9fbd34917977aae518bfd57a5" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37049 } } peers { permanent_uuid: "1879fc24146948cebd593f0c0d195642" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34997 } }
I20260806 17:06:31.209687 28307 raft_consensus.cc:374] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:31.209754 28307 raft_consensus.cc:729] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1879fc24146948cebd593f0c0d195642, State: Initialized, Role: FOLLOWER
I20260806 17:06:31.209683 28306 consensus_queue.cc:260] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956b8702e0334a36b30d3172e4515a73" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42359 } } peers { permanent_uuid: "d2e31af9fbd34917977aae518bfd57a5" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37049 } } peers { permanent_uuid: "1879fc24146948cebd593f0c0d195642" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34997 } }
I20260806 17:06:31.210207 28307 consensus_queue.cc:260] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956b8702e0334a36b30d3172e4515a73" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42359 } } peers { permanent_uuid: "d2e31af9fbd34917977aae518bfd57a5" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37049 } } peers { permanent_uuid: "1879fc24146948cebd593f0c0d195642" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34997 } }
I20260806 17:06:31.210421 28289 heartbeater.cc:503] Master 127.9.111.126:38721 was elected leader, sending a full tablet report...
I20260806 17:06:31.210752 28306 ts_tablet_manager.cc:1428] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 17:06:31.211143 28307 ts_tablet_manager.cc:1428] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 17:06:31.211540 28156 heartbeater.cc:503] Master 127.9.111.126:38721 was elected leader, sending a full tablet report...
I20260806 17:06:31.213440 28309 raft_consensus.cc:348] T c4221afb9e464063994db8737c8e67c9 P 956b8702e0334a36b30d3172e4515a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956b8702e0334a36b30d3172e4515a73" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42359 } } peers { permanent_uuid: "d2e31af9fbd34917977aae518bfd57a5" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37049 } } peers { permanent_uuid: "1879fc24146948cebd593f0c0d195642" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34997 } }
I20260806 17:06:31.213752 28309 raft_consensus.cc:374] T c4221afb9e464063994db8737c8e67c9 P 956b8702e0334a36b30d3172e4515a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:31.213857 28309 raft_consensus.cc:729] T c4221afb9e464063994db8737c8e67c9 P 956b8702e0334a36b30d3172e4515a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 956b8702e0334a36b30d3172e4515a73, State: Initialized, Role: FOLLOWER
I20260806 17:06:31.214339 28309 consensus_queue.cc:260] T c4221afb9e464063994db8737c8e67c9 P 956b8702e0334a36b30d3172e4515a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956b8702e0334a36b30d3172e4515a73" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42359 } } peers { permanent_uuid: "d2e31af9fbd34917977aae518bfd57a5" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37049 } } peers { permanent_uuid: "1879fc24146948cebd593f0c0d195642" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34997 } }
I20260806 17:06:31.215032 28023 heartbeater.cc:503] Master 127.9.111.126:38721 was elected leader, sending a full tablet report...
I20260806 17:06:31.215380 28309 ts_tablet_manager.cc:1428] T c4221afb9e464063994db8737c8e67c9 P 956b8702e0334a36b30d3172e4515a73: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260806 17:06:31.254664 28313 raft_consensus.cc:482] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:06:31.254904 28313 raft_consensus.cc:504] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956b8702e0334a36b30d3172e4515a73" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42359 } } peers { permanent_uuid: "d2e31af9fbd34917977aae518bfd57a5" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37049 } } peers { permanent_uuid: "1879fc24146948cebd593f0c0d195642" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34997 } }
I20260806 17:06:31.256090 28313 leader_election.cc:288] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 956b8702e0334a36b30d3172e4515a73 (127.9.111.65:42359), d2e31af9fbd34917977aae518bfd57a5 (127.9.111.67:37049)
I20260806 17:06:31.260208 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:31.256637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:52321 (local address 127.9.111.65:42359)
0806 17:06:31.256799 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:31.256806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:31.256830 (+ 24us) server_negotiation.cc:407] Connection header received
0806 17:06:31.256895 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:31.256898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:31.256953 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:31.257066 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:31.257375 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.257982 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.259074 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.259501 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.259650 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:31.259711 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:31.260004 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 17:06:31.260080 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 17:06:31.260208 28158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:31.256412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:42359 (local address 127.9.111.66:52321)
0806 17:06:31.256593 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:31.256609 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:31.256756 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:31.257088 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:31.257091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:31.257109 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:31.257226 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:31.257233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.258108 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.258114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:31.258926 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:31.258936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.259675 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.259678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:31.259841 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:31.259875 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 17:06:31.259972 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 17:06:31.260056 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":4}
I20260806 17:06:31.260995 27976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4221afb9e464063994db8737c8e67c9" candidate_uuid: "1879fc24146948cebd593f0c0d195642" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "956b8702e0334a36b30d3172e4515a73" is_pre_election: true
I20260806 17:06:31.261198 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:31.256707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:37049 (local address 127.9.111.66:43781)
0806 17:06:31.257087 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:31.257100 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:31.257229 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:31.257542 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:31.257545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:31.257562 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:31.257716 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:31.257723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.258647 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.258655 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:31.259798 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:31.259828 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.260718 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.260721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:31.260817 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:31.260840 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 17:06:31.260960 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 17:06:31.261043 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":68,"thread_start_us":146,"threads_started":1}
I20260806 17:06:31.261255 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:31.256870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:43781 (local address 127.9.111.67:37049)
0806 17:06:31.257011 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:31.257016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:31.257186 (+ 170us) server_negotiation.cc:407] Connection header received
0806 17:06:31.257369 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:31.257375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:31.257425 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:31.257516 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:31.257899 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.258512 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.259950 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.260555 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.260683 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:31.260736 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:31.261009 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 17:06:31.261116 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 17:06:31.261395 27976 raft_consensus.cc:2454] T c4221afb9e464063994db8737c8e67c9 P 956b8702e0334a36b30d3172e4515a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1879fc24146948cebd593f0c0d195642 in term 0.
I20260806 17:06:31.261950 28041 leader_election.cc:302] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1879fc24146948cebd593f0c0d195642, 956b8702e0334a36b30d3172e4515a73; no voters:
I20260806 17:06:31.262001 28243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4221afb9e464063994db8737c8e67c9" candidate_uuid: "1879fc24146948cebd593f0c0d195642" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2e31af9fbd34917977aae518bfd57a5" is_pre_election: true
I20260806 17:06:31.262295 28313 raft_consensus.cc:2781] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:06:31.262300 28243 raft_consensus.cc:2454] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1879fc24146948cebd593f0c0d195642 in term 0.
I20260806 17:06:31.262398 28313 raft_consensus.cc:482] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:06:31.262462 28313 raft_consensus.cc:3037] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:31.263523 28313 raft_consensus.cc:504] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956b8702e0334a36b30d3172e4515a73" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42359 } } peers { permanent_uuid: "d2e31af9fbd34917977aae518bfd57a5" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37049 } } peers { permanent_uuid: "1879fc24146948cebd593f0c0d195642" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34997 } }
I20260806 17:06:31.263880 28313 leader_election.cc:288] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [CANDIDATE]: Term 1 election: Requested vote from peers 956b8702e0334a36b30d3172e4515a73 (127.9.111.65:42359), d2e31af9fbd34917977aae518bfd57a5 (127.9.111.67:37049)
I20260806 17:06:31.264268 27976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4221afb9e464063994db8737c8e67c9" candidate_uuid: "1879fc24146948cebd593f0c0d195642" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "956b8702e0334a36b30d3172e4515a73"
I20260806 17:06:31.264400 28243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4221afb9e464063994db8737c8e67c9" candidate_uuid: "1879fc24146948cebd593f0c0d195642" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2e31af9fbd34917977aae518bfd57a5"
I20260806 17:06:31.264407 27976 raft_consensus.cc:3037] T c4221afb9e464063994db8737c8e67c9 P 956b8702e0334a36b30d3172e4515a73 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:31.264551 28243 raft_consensus.cc:3037] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:31.265578 27976 raft_consensus.cc:2454] T c4221afb9e464063994db8737c8e67c9 P 956b8702e0334a36b30d3172e4515a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1879fc24146948cebd593f0c0d195642 in term 1.
I20260806 17:06:31.265671 28243 raft_consensus.cc:2454] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1879fc24146948cebd593f0c0d195642 in term 1.
I20260806 17:06:31.265937 28041 leader_election.cc:302] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1879fc24146948cebd593f0c0d195642, 956b8702e0334a36b30d3172e4515a73; no voters:
I20260806 17:06:31.266121 28313 raft_consensus.cc:2781] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:31.266368 28313 raft_consensus.cc:686] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [term 1 LEADER]: Becoming Leader. State: Replica: 1879fc24146948cebd593f0c0d195642, State: Running, Role: LEADER
I20260806 17:06:31.266902 28313 consensus_queue.cc:237] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956b8702e0334a36b30d3172e4515a73" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42359 } } peers { permanent_uuid: "d2e31af9fbd34917977aae518bfd57a5" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37049 } } peers { permanent_uuid: "1879fc24146948cebd593f0c0d195642" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34997 } }
I20260806 17:06:31.270318 27832 catalog_manager.cc:5495] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1879fc24146948cebd593f0c0d195642 (127.9.111.66). New cstate: current_term: 1 leader_uuid: "1879fc24146948cebd593f0c0d195642" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956b8702e0334a36b30d3172e4515a73" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2e31af9fbd34917977aae518bfd57a5" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1879fc24146948cebd593f0c0d195642" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34997 } health_report { overall_health: HEALTHY } } }
I20260806 17:06:31.289155 28305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:31.285617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52820 (local address 127.9.111.66:34997)
0806 17:06:31.285757 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:31.285764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:31.285779 (+ 15us) server_negotiation.cc:407] Connection header received
0806 17:06:31.285879 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:31.285884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:31.285943 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:31.286054 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:06:31.286317 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.286823 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.287658 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:31.287885 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:31.287992 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:31.288338 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:06:31.288762 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:06:31.288887 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:31.288960 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 17:06:31.289010 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 17:06:31.295190 27976 raft_consensus.cc:1261] T c4221afb9e464063994db8737c8e67c9 P 956b8702e0334a36b30d3172e4515a73 [term 1 FOLLOWER]: Refusing update from remote peer 1879fc24146948cebd593f0c0d195642: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:06:31.295190 28243 raft_consensus.cc:1261] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5 [term 1 FOLLOWER]: Refusing update from remote peer 1879fc24146948cebd593f0c0d195642: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:06:31.296020 28313 consensus_queue.cc:1035] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2e31af9fbd34917977aae518bfd57a5" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:06:31.296218 28316 consensus_queue.cc:1035] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956b8702e0334a36b30d3172e4515a73" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 42359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:06:31.307240 28321 mvcc.cc:205] Tried to move back new op lower bound from 7315603420337164288 to 7315603420232372224. Current Snapshot: MvccSnapshot[applied={T|T < 7315603420337164288}]
I20260806 17:06:31.313174 27832 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:06:31.313390 27832 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:06:31.316210 27832 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34f90341a28842438950bb7602804531: Sending DeleteTablet for 3 replicas of tablet c4221afb9e464063994db8737c8e67c9
I20260806 17:06:31.317066 27956 tablet_service.cc:1514] Processing DeleteTablet for tablet c4221afb9e464063994db8737c8e67c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:06:31 UTC) from {username='slave'} at 127.0.0.1:45692
I20260806 17:06:31.317394 28222 tablet_service.cc:1514] Processing DeleteTablet for tablet c4221afb9e464063994db8737c8e67c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:06:31 UTC) from {username='slave'} at 127.0.0.1:46888
I20260806 17:06:31.317623 28329 tablet_replica.cc:330] T c4221afb9e464063994db8737c8e67c9 P 956b8702e0334a36b30d3172e4515a73: stopping tablet replica
I20260806 17:06:31.317792 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 27893
I20260806 17:06:31.318038 28329 raft_consensus.cc:2229] T c4221afb9e464063994db8737c8e67c9 P 956b8702e0334a36b30d3172e4515a73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:06:31.318022 28090 tablet_service.cc:1514] Processing DeleteTablet for tablet c4221afb9e464063994db8737c8e67c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:06:31 UTC) from {username='slave'} at 127.0.0.1:52808
I20260806 17:06:31.319870 28331 tablet_replica.cc:330] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642: stopping tablet replica
I20260806 17:06:31.320194 28331 raft_consensus.cc:2229] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:06:31.320662 28331 raft_consensus.cc:2258] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:06:31.321697 28330 tablet_replica.cc:330] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5: stopping tablet replica
I20260806 17:06:31.322429 28330 raft_consensus.cc:2229] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:06:31.322754 28330 raft_consensus.cc:2258] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:06:31.323240 28331 ts_tablet_manager.cc:1905] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:06:31.324174 28330 ts_tablet_manager.cc:1905] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:06:31.326385 28331 ts_tablet_manager.cc:1918] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:06:31.326565 28331 log.cc:1196] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/c4221afb9e464063994db8737c8e67c9
I20260806 17:06:31.327065 28331 ts_tablet_manager.cc:1939] T c4221afb9e464063994db8737c8e67c9 P 1879fc24146948cebd593f0c0d195642: Deleting consensus metadata
I20260806 17:06:31.327188 28330 ts_tablet_manager.cc:1918] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:06:31.327296 28330 log.cc:1196] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/c4221afb9e464063994db8737c8e67c9
I20260806 17:06:31.327606 28330 ts_tablet_manager.cc:1939] T c4221afb9e464063994db8737c8e67c9 P d2e31af9fbd34917977aae518bfd57a5: Deleting consensus metadata
W20260806 17:06:31.328044 27815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.111.65:42359 (error 108)
I20260806 17:06:31.328742 27815 catalog_manager.cc:4841] TS d2e31af9fbd34917977aae518bfd57a5 (127.9.111.67:37049): tablet c4221afb9e464063994db8737c8e67c9 (table test-table [id=93b238d27b2945ceaa436c6d5b220239]) successfully deleted
I20260806 17:06:31.328994 27815 catalog_manager.cc:4841] TS 1879fc24146948cebd593f0c0d195642 (127.9.111.66:34997): tablet c4221afb9e464063994db8737c8e67c9 (table test-table [id=93b238d27b2945ceaa436c6d5b220239]) successfully deleted
I20260806 17:06:31.329672 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 28026
I20260806 17:06:31.329790 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:31.329359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:42359 (local address 127.0.0.1:45704)
0806 17:06:31.329589 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:31.329684 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.65:42359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260806 17:06:31.333072 27815 catalog_manager.cc:4584] TS 956b8702e0334a36b30d3172e4515a73 (127.9.111.65:42359): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c4221afb9e464063994db8737c8e67c9: Network error: Client connection negotiation failed: client connection to 127.9.111.65:42359: connect: Connection refused (error 111)
I20260806 17:06:31.336982 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 28159
I20260806 17:06:31.343372 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 27800
2026-08-06T17:06:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1164 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:06:31 dist-test-slave-69rg krb5kdc[28336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:06:31 dist-test-slave-69rg krb5kdc[28336](info): set up 2 sockets
Aug 06 17:06:31 dist-test-slave-69rg krb5kdc[28336](info): commencing operation
krb5kdc: starting...
W20260806 17:06:32.422150 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:06:32 dist-test-slave-69rg krb5kdc[28336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:06:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:06:32.556607 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:35385
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:35385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:32.671002 28349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:32.671356 28349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:32.671439 28349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:32.675509 28349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:32.675608 28349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:32.675647 28349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:32.675685 28349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:32.675720 28349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:32.681103 28349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:35385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:35385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:32.682348 28349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:32.683403 28349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:32.692099 28354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:32.692086 28357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:32.692168 28349 server_base.cc:1034] running on GCE node
W20260806 17:06:32.692099 28355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:32.692945 28349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:32.694213 28349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:32.695395 28349 hybrid_clock.cc:648] HybridClock initialized: now 1786035992695374 us; error 41 us; skew 500 ppm
Aug 06 17:06:32 dist-test-slave-69rg krb5kdc[28336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035992, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:32.698356 28349 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:06:32.699242 28349 webserver.cc:460] Webserver started at http://127.9.111.126:41583/ using document root <none> and password file <none>
I20260806 17:06:32.699829 28349 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:32.699918 28349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:32.700135 28349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:32.702273 28349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "e4fef6c94ed949d79df815bd1908279b"
format_stamp: "Formatted at 2026-08-06 17:06:32 on dist-test-slave-69rg"
server_key: "1c41db8f45e734867f932e8d6d4aa7e0"
server_key_iv: "8e46d9ec5ccdc39b62e92e172f0c25c3"
server_key_version: "encryptionkey@0"
I20260806 17:06:32.702867 28349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "e4fef6c94ed949d79df815bd1908279b"
format_stamp: "Formatted at 2026-08-06 17:06:32 on dist-test-slave-69rg"
server_key: "1c41db8f45e734867f932e8d6d4aa7e0"
server_key_iv: "8e46d9ec5ccdc39b62e92e172f0c25c3"
server_key_version: "encryptionkey@0"
I20260806 17:06:32.706722 28349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 17:06:32.709352 28364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:32.710796 28349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:06:32.710960 28349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "e4fef6c94ed949d79df815bd1908279b"
format_stamp: "Formatted at 2026-08-06 17:06:32 on dist-test-slave-69rg"
server_key: "1c41db8f45e734867f932e8d6d4aa7e0"
server_key_iv: "8e46d9ec5ccdc39b62e92e172f0c25c3"
server_key_version: "encryptionkey@0"
I20260806 17:06:32.711094 28349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:32.729327 28349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:32.732944 28349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:32.733166 28349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:32.740835 28349 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:35385
I20260806 17:06:32.740862 28416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:35385 every 8 connection(s)
I20260806 17:06:32.741945 28349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:32.744279 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 28349
I20260806 17:06:32.744429 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:32.744768 9661 external_mini_cluster.cc:1428] Setting key 366bf1a56fcd1eac55b904a747608dca
I20260806 17:06:32.745072 28417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:32.751461 28417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b: Bootstrap starting.
I20260806 17:06:32.753754 28417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b: Neither blocks nor log segments found. Creating new log.
Aug 06 17:06:32 dist-test-slave-69rg krb5kdc[28336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:06:32.754714 28417 log.cc:824] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:32.756767 28417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b: No bootstrap required, opened a new log
I20260806 17:06:32.759760 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:32.746296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60756 (local address 127.9.111.126:35385)
0806 17:06:32.746773 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:32.746789 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:32.746824 (+ 35us) server_negotiation.cc:407] Connection header received
0806 17:06:32.747856 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:32.747874 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:32.748201 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:32.748538 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:32.749398 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:32.750491 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:32.751365 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:32.751903 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:32.752075 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:32.754949 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:32.754955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:32.754968 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:32.754996 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:32.756802 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:32.757287 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:32.757295 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:32.757305 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:32.757418 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:32.757718 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:32.757721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:32.757723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:32.758199 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:32.758418 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:32.758725 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 17:06:32.759023 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":145,"threads_started":1}
I20260806 17:06:32.760557 28417 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4fef6c94ed949d79df815bd1908279b" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 35385 } }
I20260806 17:06:32.760847 28417 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:32.760903 28417 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4fef6c94ed949d79df815bd1908279b, State: Initialized, Role: FOLLOWER
I20260806 17:06:32.761363 28417 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4fef6c94ed949d79df815bd1908279b" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 35385 } }
I20260806 17:06:32.761518 28417 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:32.761583 28417 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:32.761677 28417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:32.762814 28417 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4fef6c94ed949d79df815bd1908279b" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 35385 } }
I20260806 17:06:32.763156 28417 leader_election.cc:302] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4fef6c94ed949d79df815bd1908279b; no voters:
I20260806 17:06:32.763533 28417 leader_election.cc:288] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:32.763746 28422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:32.764040 28422 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b [term 1 LEADER]: Becoming Leader. State: Replica: e4fef6c94ed949d79df815bd1908279b, State: Running, Role: LEADER
I20260806 17:06:32.764344 28422 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4fef6c94ed949d79df815bd1908279b" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 35385 } }
I20260806 17:06:32.764917 28417 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:32.766165 28424 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4fef6c94ed949d79df815bd1908279b. Latest consensus state: current_term: 1 leader_uuid: "e4fef6c94ed949d79df815bd1908279b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4fef6c94ed949d79df815bd1908279b" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 35385 } } }
I20260806 17:06:32.766312 28424 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:32.766739 28431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:32.766701 28423 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4fef6c94ed949d79df815bd1908279b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4fef6c94ed949d79df815bd1908279b" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 35385 } } }
I20260806 17:06:32.767513 28423 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:32.770116 28431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:32.776024 28431 catalog_manager.cc:1347] Generated new cluster ID: c4766560e6884085a438ebb685ab15cc
I20260806 17:06:32.776103 28431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:32.785049 28431 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:32.786142 28431 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:32.801174 28431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b: Generated new TSK 0
I20260806 17:06:32.802084 28431 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
I20260806 17:06:32.845371 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:35385
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:32.960990 28443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:32.961288 28443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:32.961336 28443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:32.965790 28443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:32.965879 28443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:32.965981 28443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:06:32.972182 28443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:32.973459 28443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:32.974699 28443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:32.984808 28451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:32.984812 28448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:32.984781 28449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:32.985585 28443 server_base.cc:1034] running on GCE node
I20260806 17:06:32.986014 28443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:32.986671 28443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:32.987839 28443 hybrid_clock.cc:648] HybridClock initialized: now 1786035992987827 us; error 26 us; skew 500 ppm
Aug 06 17:06:32 dist-test-slave-69rg krb5kdc[28336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035992, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:32.990993 28443 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:06:32.992143 28443 webserver.cc:460] Webserver started at http://127.9.111.65:38517/ using document root <none> and password file <none>
I20260806 17:06:32.992808 28443 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:32.992913 28443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:32.993187 28443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:32.995625 28443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "2cb03c7c09434874955b88752a3b185e"
format_stamp: "Formatted at 2026-08-06 17:06:32 on dist-test-slave-69rg"
server_key: "4d0b2c11d90c0368edd858784fbc2744"
server_key_iv: "44438fdc8cb05eb436cf692444b45361"
server_key_version: "encryptionkey@0"
I20260806 17:06:32.996280 28443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "2cb03c7c09434874955b88752a3b185e"
format_stamp: "Formatted at 2026-08-06 17:06:32 on dist-test-slave-69rg"
server_key: "4d0b2c11d90c0368edd858784fbc2744"
server_key_iv: "44438fdc8cb05eb436cf692444b45361"
server_key_version: "encryptionkey@0"
I20260806 17:06:33.000566 28443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260806 17:06:33.003743 28458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:33.005169 28443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 17:06:33.005336 28443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "2cb03c7c09434874955b88752a3b185e"
format_stamp: "Formatted at 2026-08-06 17:06:32 on dist-test-slave-69rg"
server_key: "4d0b2c11d90c0368edd858784fbc2744"
server_key_iv: "44438fdc8cb05eb436cf692444b45361"
server_key_version: "encryptionkey@0"
I20260806 17:06:33.005476 28443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:33.027704 28443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:33.030988 28443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:33.031213 28443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:33.031836 28443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:33.032701 28443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:33.032778 28443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:33.032848 28443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:33.032898 28443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:33.042070 28443 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:40027
I20260806 17:06:33.042101 28571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:40027 every 8 connection(s)
I20260806 17:06:33.042972 28443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:06:33.043388 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 28443
I20260806 17:06:33.043540 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:06:33.043879 9661 external_mini_cluster.cc:1428] Setting key 6721063bf3262942c7f2725265960d6e
Aug 06 17:06:33 dist-test-slave-69rg krb5kdc[28336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035992, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.111.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.66@KRBTEST.COM" created.
I20260806 17:06:33.060352 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:33.044780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:47625 (local address 127.9.111.126:35385)
0806 17:06:33.044933 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:33.044938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:33.046069 (+ 1131us) server_negotiation.cc:407] Connection header received
0806 17:06:33.048084 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:33.048090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:33.048163 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:33.048275 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:33.049952 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.050518 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.052454 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.052718 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.052802 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:33.055925 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:33.055929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:33.055931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:33.055961 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:33.057733 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:33.058425 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:33.058429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:33.058431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:33.058532 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:33.058945 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:33.058949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:33.058951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:33.059294 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:33.059404 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:33.059982 (+ 578us) server_negotiation.cc:299] Negotiation successful
0806 17:06:33.060149 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 17:06:33.061264 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:33.045328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:35385 (local address 127.9.111.65:47625)
0806 17:06:33.045890 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:33.045955 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:33.047770 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:33.048447 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:33.048454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:33.048972 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:33.049674 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:33.049697 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.050672 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.050679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:33.052294 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:33.052310 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.052895 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.052901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:33.053147 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:33.053863 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:06:33.053888 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:33.055683 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:33.057942 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:33.057956 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:33.057971 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:33.058297 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:33.058654 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:33.058658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:33.058660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:33.058833 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:33.059404 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:33.059413 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:33.059711 (+ 298us) client_negotiation.cc:769] Sending connection context
0806 17:06:33.059933 (+ 222us) client_negotiation.cc:240] Negotiation successful
0806 17:06:33.060316 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":112,"threads_started":1}
I20260806 17:06:33.062933 28572 heartbeater.cc:347] Connected to a master server at 127.9.111.126:35385
I20260806 17:06:33.063380 28572 heartbeater.cc:464] Registering TS with master...
I20260806 17:06:33.063952 28572 heartbeater.cc:511] Master 127.9.111.126:35385 requested a full tablet report, sending...
I20260806 17:06:33.065696 28381 ts_manager.cc:194] Registered new tserver with Master: 2cb03c7c09434874955b88752a3b185e (127.9.111.65:40027)
I20260806 17:06:33.066939 28381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:47625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
I20260806 17:06:33.077615 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:35385
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:33.191918 28577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:33.192207 28577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:33.192313 28577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:33.196508 28577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:33.196574 28577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:33.196678 28577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:06:33.202034 28577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:33.203195 28577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:33.204249 28577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:33.214156 28582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:33.214156 28583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:33.214287 28577 server_base.cc:1034] running on GCE node
W20260806 17:06:33.214160 28585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:33.214890 28577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:33.215546 28577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:33.216745 28577 hybrid_clock.cc:648] HybridClock initialized: now 1786035993216742 us; error 72 us; skew 500 ppm
Aug 06 17:06:33 dist-test-slave-69rg krb5kdc[28336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035993, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:33.219764 28577 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:06:33.220785 28577 webserver.cc:460] Webserver started at http://127.9.111.66:40959/ using document root <none> and password file <none>
I20260806 17:06:33.221412 28577 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:33.221513 28577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:33.221778 28577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:33.224216 28577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "0d615055377c4a95a1355a80e3894009"
format_stamp: "Formatted at 2026-08-06 17:06:33 on dist-test-slave-69rg"
server_key: "83844dd7b86ec2f63b68cc0938520f11"
server_key_iv: "bec44e102b50aed0b89ed7331b0f20fa"
server_key_version: "encryptionkey@0"
I20260806 17:06:33.224862 28577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "0d615055377c4a95a1355a80e3894009"
format_stamp: "Formatted at 2026-08-06 17:06:33 on dist-test-slave-69rg"
server_key: "83844dd7b86ec2f63b68cc0938520f11"
server_key_iv: "bec44e102b50aed0b89ed7331b0f20fa"
server_key_version: "encryptionkey@0"
I20260806 17:06:33.229120 28577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260806 17:06:33.231806 28592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:33.233027 28577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 17:06:33.233178 28577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "0d615055377c4a95a1355a80e3894009"
format_stamp: "Formatted at 2026-08-06 17:06:33 on dist-test-slave-69rg"
server_key: "83844dd7b86ec2f63b68cc0938520f11"
server_key_iv: "bec44e102b50aed0b89ed7331b0f20fa"
server_key_version: "encryptionkey@0"
I20260806 17:06:33.233306 28577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:33.257294 28577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:33.260818 28577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:33.261047 28577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:33.261685 28577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:33.262619 28577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:33.262730 28577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:33.262831 28577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:33.262888 28577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:33.273070 28577 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:33005
I20260806 17:06:33.273097 28705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:33005 every 8 connection(s)
I20260806 17:06:33.274108 28577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:06:33.275926 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 28577
I20260806 17:06:33.276077 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:06:33.276404 9661 external_mini_cluster.cc:1428] Setting key a9ae67fd9244e8dc1142e6231278253b
Aug 06 17:06:33 dist-test-slave-69rg krb5kdc[28336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035993, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.111.67@KRBTEST.COM; defaulting to no policy
I20260806 17:06:33.288880 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:33.276051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:49127 (local address 127.9.111.126:35385)
0806 17:06:33.276211 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:33.276216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:33.277171 (+ 955us) server_negotiation.cc:407] Connection header received
0806 17:06:33.278584 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:33.278590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:33.278655 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:33.278751 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:33.279946 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.280502 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.281644 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.281955 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.282069 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:33.285000 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:33.285006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:33.285009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:33.285043 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:33.286795 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:33.287326 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:33.287329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:33.287330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:33.287409 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:33.287749 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:33.287753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:33.287754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:33.288020 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:33.288122 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:33.288643 (+ 521us) server_negotiation.cc:299] Negotiation successful
0806 17:06:33.288757 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.67@KRBTEST.COM" created.
I20260806 17:06:33.289872 28708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:33.276457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:35385 (local address 127.9.111.66:49127)
0806 17:06:33.276974 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:33.277027 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:33.278380 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:33.278904 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:33.278910 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:33.279327 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:33.279794 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:33.279808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.280634 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.280638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:33.281505 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:33.281514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.282119 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.282124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:33.282369 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:33.282992 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:06:33.283008 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:33.284813 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:33.286939 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:33.286945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:33.286966 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:33.287220 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:33.287494 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:33.287497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:33.287498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:33.287636 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:33.288135 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:33.288140 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:33.288394 (+ 254us) client_negotiation.cc:769] Sending connection context
0806 17:06:33.288554 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 17:06:33.288815 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":141,"threads_started":1}
I20260806 17:06:33.291241 28706 heartbeater.cc:347] Connected to a master server at 127.9.111.126:35385
I20260806 17:06:33.291497 28706 heartbeater.cc:464] Registering TS with master...
I20260806 17:06:33.292137 28706 heartbeater.cc:511] Master 127.9.111.126:35385 requested a full tablet report, sending...
I20260806 17:06:33.293383 28381 ts_manager.cc:194] Registered new tserver with Master: 0d615055377c4a95a1355a80e3894009 (127.9.111.66:33005)
I20260806 17:06:33.294282 28381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:49127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
I20260806 17:06:33.309101 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:35385
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:33.427754 28711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:33.428140 28711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:33.428233 28711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:33.432894 28711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:33.433027 28711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:33.433187 28711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:06:33.439843 28711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:33.441193 28711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:33.442483 28711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:33.453536 28719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:33.453536 28716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:33.453644 28711 server_base.cc:1034] running on GCE node
W20260806 17:06:33.453584 28717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:33.454401 28711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:33.455107 28711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:33.456310 28711 hybrid_clock.cc:648] HybridClock initialized: now 1786035993456290 us; error 33 us; skew 500 ppm
Aug 06 17:06:33 dist-test-slave-69rg krb5kdc[28336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035993, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:33.459288 28711 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:06:33.460306 28711 webserver.cc:460] Webserver started at http://127.9.111.67:44231/ using document root <none> and password file <none>
I20260806 17:06:33.460914 28711 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:33.461014 28711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:33.461268 28711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:33.463480 28711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "5e1ea21308f84a6792d3cc4e280ae306"
format_stamp: "Formatted at 2026-08-06 17:06:33 on dist-test-slave-69rg"
server_key: "0d9a624a63fd8206888f6b91cec355a6"
server_key_iv: "78b82dd2ab83e3d8dabafe34e2ead69a"
server_key_version: "encryptionkey@0"
I20260806 17:06:33.464113 28711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "5e1ea21308f84a6792d3cc4e280ae306"
format_stamp: "Formatted at 2026-08-06 17:06:33 on dist-test-slave-69rg"
server_key: "0d9a624a63fd8206888f6b91cec355a6"
server_key_iv: "78b82dd2ab83e3d8dabafe34e2ead69a"
server_key_version: "encryptionkey@0"
I20260806 17:06:33.468251 28711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 17:06:33.470700 28726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:33.471909 28711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:06:33.472045 28711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "5e1ea21308f84a6792d3cc4e280ae306"
format_stamp: "Formatted at 2026-08-06 17:06:33 on dist-test-slave-69rg"
server_key: "0d9a624a63fd8206888f6b91cec355a6"
server_key_iv: "78b82dd2ab83e3d8dabafe34e2ead69a"
server_key_version: "encryptionkey@0"
I20260806 17:06:33.472167 28711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:33.483989 28711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:33.487210 28711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:33.487422 28711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:33.488041 28711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:33.488893 28711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:33.488982 28711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:33.489074 28711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:33.489130 28711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:33.498992 28711 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:41617
I20260806 17:06:33.499019 28840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:41617 every 8 connection(s)
I20260806 17:06:33.499926 28711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:06:33.507283 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 28711
I20260806 17:06:33.507432 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:06:33.507774 9661 external_mini_cluster.cc:1428] Setting key 27b0486049d7a82ca2a541bbe4e97f8c
Aug 06 17:06:33 dist-test-slave-69rg krb5kdc[28336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035993, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:06:33.514293 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:33.501812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:54841 (local address 127.9.111.126:35385)
0806 17:06:33.501975 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:33.501979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:33.502780 (+ 801us) server_negotiation.cc:407] Connection header received
0806 17:06:33.504104 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:33.504107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:33.504171 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:33.504275 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:33.505446 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.506084 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.507240 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.507566 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.507690 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:33.510774 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:33.510777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:33.510779 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:33.510804 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:33.512158 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:33.512672 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:33.512675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:33.512676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:33.512754 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:33.513120 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:33.513123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:33.513124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:33.513310 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:33.513388 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:33.513952 (+ 564us) server_negotiation.cc:299] Negotiation successful
0806 17:06:33.514124 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 17:06:33.515264 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:33.502179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:35385 (local address 127.9.111.67:54841)
0806 17:06:33.502620 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:33.502666 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:33.503882 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:33.504451 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:33.504458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:33.504813 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:33.505294 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:33.505307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.506213 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.506217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:33.507092 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:33.507102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.507731 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.507735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:33.507939 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:33.508611 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:06:33.508660 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:33.510616 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:33.512275 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:33.512280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:33.512292 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:33.512554 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:33.512875 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:33.512878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:33.512879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:33.513027 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:33.513407 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:33.513413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:33.513653 (+ 240us) client_negotiation.cc:769] Sending connection context
0806 17:06:33.513888 (+ 235us) client_negotiation.cc:240] Negotiation successful
0806 17:06:33.514314 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":105,"threads_started":1}
I20260806 17:06:33.516649 28841 heartbeater.cc:347] Connected to a master server at 127.9.111.126:35385
I20260806 17:06:33.516920 28841 heartbeater.cc:464] Registering TS with master...
I20260806 17:06:33.517445 28841 heartbeater.cc:511] Master 127.9.111.126:35385 requested a full tablet report, sending...
I20260806 17:06:33.518633 28381 ts_manager.cc:194] Registered new tserver with Master: 5e1ea21308f84a6792d3cc4e280ae306 (127.9.111.67:41617)
I20260806 17:06:33.519302 28381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:54841
I20260806 17:06:33.522341 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 17:06:33.532622 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:33.524369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60766 (local address 127.9.111.126:35385)
0806 17:06:33.524536 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:33.524541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:33.524698 (+ 157us) server_negotiation.cc:407] Connection header received
0806 17:06:33.524854 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:33.524857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:33.524907 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:33.525021 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:33.525730 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.526417 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.527269 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.527505 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.527596 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:33.529187 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:33.529190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:33.529192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:33.529218 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:33.530923 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:33.531432 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:33.531435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:33.531443 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:33.531527 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:33.531831 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:33.531833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:33.531834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:33.532048 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:33.532180 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:33.532404 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 17:06:33.532491 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 17:06:33.535761 28381 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:06:33.539265 28381 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:06:33.554250 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:33.547891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.9.111.67:41617)
0806 17:06:33.548942 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:33.548955 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:33.548973 (+ 18us) server_negotiation.cc:407] Connection header received
0806 17:06:33.549049 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:33.549054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:33.549256 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:33.549803 (+ 547us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:33.549982 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.551071 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.552884 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.553443 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.553582 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:33.553753 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:33.553987 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 17:06:33.554086 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":944,"thread_start_us":182,"threads_started":1}
I20260806 17:06:33.554240 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:33.547746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:41617 (local address 127.0.0.1:39684)
0806 17:06:33.548076 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:33.548130 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:33.548253 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:33.549438 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:33.549444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:33.549474 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:33.549637 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:33.549643 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.551230 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.551233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:33.552737 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:33.552748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.553586 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.553589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:33.553712 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:33.553749 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 17:06:33.553899 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 17:06:33.554053 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":103,"thread_start_us":72,"threads_started":1}
I20260806 17:06:33.554585 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:33.549256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58340 (local address 127.9.111.65:40027)
0806 17:06:33.550014 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:33.550027 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:33.550045 (+ 18us) server_negotiation.cc:407] Connection header received
0806 17:06:33.550103 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:33.550106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:33.550241 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:33.550393 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:33.550591 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.551467 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.552868 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.553635 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.553738 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:33.553897 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:33.554322 (+ 425us) server_negotiation.cc:299] Negotiation successful
0806 17:06:33.554435 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":103,"threads_started":1}
I20260806 17:06:33.554251 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:33.548826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:40027 (local address 127.0.0.1:58340)
0806 17:06:33.549662 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:33.549676 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:33.549767 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:33.550372 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:33.550375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:33.550393 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:33.550492 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:33.550498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.551618 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.551621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:33.552748 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:33.552755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.553907 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.553910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:33.554005 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:33.554033 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 17:06:33.554100 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 17:06:33.554160 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":26,"thread_start_us":98,"threads_started":1}
I20260806 17:06:33.555152 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:33.548488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51156 (local address 127.9.111.66:33005)
0806 17:06:33.548933 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:33.548947 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:33.548973 (+ 26us) server_negotiation.cc:407] Connection header received
0806 17:06:33.549049 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:33.549054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:33.549256 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:33.549419 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:33.549776 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.551031 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.552803 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.553605 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.554072 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:33.554198 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:33.554246 (+ 48us) server_negotiation.cc:299] Negotiation successful
0806 17:06:33.554338 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":130,"threads_started":1}
I20260806 17:06:33.555493 28853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:33.547907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:33005 (local address 127.0.0.1:51156)
0806 17:06:33.548573 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:33.548590 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:33.548697 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:33.549438 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:33.549444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:33.549485 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:33.549637 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:33.549645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.551174 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.551178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:33.552681 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:33.552691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.553760 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.553764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:33.553890 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:33.553917 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 17:06:33.553973 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 17:06:33.555366 (+ 1393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":1305,"spinlock_wait_cycles":15488,"thread_start_us":95,"threads_started":1}
I20260806 17:06:33.557324 28774 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4da25eef084b1a9ab58dad7c48688b (DEFAULT_TABLE table=test-table [id=0fa9e0fdd79d40c1ab9939714453e0cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:06:33.557399 28506 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4da25eef084b1a9ab58dad7c48688b (DEFAULT_TABLE table=test-table [id=0fa9e0fdd79d40c1ab9939714453e0cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:06:33.558465 28774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4da25eef084b1a9ab58dad7c48688b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:33.558071 28640 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4da25eef084b1a9ab58dad7c48688b (DEFAULT_TABLE table=test-table [id=0fa9e0fdd79d40c1ab9939714453e0cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:06:33.558508 28506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4da25eef084b1a9ab58dad7c48688b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:33.559007 28640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4da25eef084b1a9ab58dad7c48688b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:33.564858 28858 tablet_bootstrap.cc:492] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306: Bootstrap starting.
I20260806 17:06:33.566406 28859 tablet_bootstrap.cc:492] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e: Bootstrap starting.
I20260806 17:06:33.566879 28860 tablet_bootstrap.cc:492] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009: Bootstrap starting.
I20260806 17:06:33.567281 28858 tablet_bootstrap.cc:654] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:33.568308 28858 log.cc:824] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:33.568694 28860 tablet_bootstrap.cc:654] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:33.569904 28860 log.cc:824] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:33.570555 28858 tablet_bootstrap.cc:492] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306: No bootstrap required, opened a new log
I20260806 17:06:33.570631 28859 tablet_bootstrap.cc:654] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:33.570802 28858 ts_tablet_manager.cc:1397] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 17:06:33.571426 28859 log.cc:824] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:33.571975 28860 tablet_bootstrap.cc:492] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009: No bootstrap required, opened a new log
I20260806 17:06:33.572217 28860 ts_tablet_manager.cc:1397] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 17:06:33.573426 28859 tablet_bootstrap.cc:492] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e: No bootstrap required, opened a new log
I20260806 17:06:33.573603 28859 ts_tablet_manager.cc:1397] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 17:06:33.574244 28858 raft_consensus.cc:348] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1ea21308f84a6792d3cc4e280ae306" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41617 } } peers { permanent_uuid: "0d615055377c4a95a1355a80e3894009" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33005 } } peers { permanent_uuid: "2cb03c7c09434874955b88752a3b185e" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40027 } }
I20260806 17:06:33.574499 28858 raft_consensus.cc:374] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:33.574553 28858 raft_consensus.cc:729] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1ea21308f84a6792d3cc4e280ae306, State: Initialized, Role: FOLLOWER
I20260806 17:06:33.574986 28858 consensus_queue.cc:260] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1ea21308f84a6792d3cc4e280ae306" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41617 } } peers { permanent_uuid: "0d615055377c4a95a1355a80e3894009" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33005 } } peers { permanent_uuid: "2cb03c7c09434874955b88752a3b185e" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40027 } }
I20260806 17:06:33.575640 28841 heartbeater.cc:503] Master 127.9.111.126:35385 was elected leader, sending a full tablet report...
I20260806 17:06:33.576005 28858 ts_tablet_manager.cc:1428] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 17:06:33.578004 28859 raft_consensus.cc:348] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1ea21308f84a6792d3cc4e280ae306" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41617 } } peers { permanent_uuid: "0d615055377c4a95a1355a80e3894009" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33005 } } peers { permanent_uuid: "2cb03c7c09434874955b88752a3b185e" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40027 } }
I20260806 17:06:33.578325 28859 raft_consensus.cc:374] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:33.578378 28859 raft_consensus.cc:729] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb03c7c09434874955b88752a3b185e, State: Initialized, Role: FOLLOWER
I20260806 17:06:33.578855 28859 consensus_queue.cc:260] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1ea21308f84a6792d3cc4e280ae306" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41617 } } peers { permanent_uuid: "0d615055377c4a95a1355a80e3894009" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33005 } } peers { permanent_uuid: "2cb03c7c09434874955b88752a3b185e" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40027 } }
I20260806 17:06:33.576503 28860 raft_consensus.cc:348] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1ea21308f84a6792d3cc4e280ae306" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41617 } } peers { permanent_uuid: "0d615055377c4a95a1355a80e3894009" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33005 } } peers { permanent_uuid: "2cb03c7c09434874955b88752a3b185e" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40027 } }
I20260806 17:06:33.579133 28860 raft_consensus.cc:374] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:33.579193 28860 raft_consensus.cc:729] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d615055377c4a95a1355a80e3894009, State: Initialized, Role: FOLLOWER
I20260806 17:06:33.579591 28860 consensus_queue.cc:260] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1ea21308f84a6792d3cc4e280ae306" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41617 } } peers { permanent_uuid: "0d615055377c4a95a1355a80e3894009" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33005 } } peers { permanent_uuid: "2cb03c7c09434874955b88752a3b185e" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40027 } }
I20260806 17:06:33.579923 28859 ts_tablet_manager.cc:1428] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 17:06:33.580258 28572 heartbeater.cc:503] Master 127.9.111.126:35385 was elected leader, sending a full tablet report...
I20260806 17:06:33.580600 28706 heartbeater.cc:503] Master 127.9.111.126:35385 was elected leader, sending a full tablet report...
I20260806 17:06:33.581323 28860 ts_tablet_manager.cc:1428] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
W20260806 17:06:33.750923 28842 tablet.cc:2369] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 17:06:33.775398 28707 tablet.cc:2369] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:06:33.775841 28865 raft_consensus.cc:482] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:06:33.776087 28865 raft_consensus.cc:504] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1ea21308f84a6792d3cc4e280ae306" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41617 } } peers { permanent_uuid: "0d615055377c4a95a1355a80e3894009" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33005 } } peers { permanent_uuid: "2cb03c7c09434874955b88752a3b185e" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40027 } }
I20260806 17:06:33.777403 28865 leader_election.cc:288] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1ea21308f84a6792d3cc4e280ae306 (127.9.111.67:41617), 0d615055377c4a95a1355a80e3894009 (127.9.111.66:33005)
I20260806 17:06:33.782064 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:33.777765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:43109 (local address 127.9.111.67:41617)
0806 17:06:33.777929 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:33.777934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:33.778377 (+ 443us) server_negotiation.cc:407] Connection header received
0806 17:06:33.778543 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:33.778545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:33.778588 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:33.778696 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:33.779035 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.779548 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.780880 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.781323 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.781457 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:33.781522 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:33.781834 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 17:06:33.781925 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 17:06:33.782130 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:33.777804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:33583 (local address 127.9.111.66:33005)
0806 17:06:33.777948 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:33.777954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:33.778265 (+ 311us) server_negotiation.cc:407] Connection header received
0806 17:06:33.778481 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:33.778487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:33.778553 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:33.778618 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:33.778966 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.779542 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.780833 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.781447 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.781579 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:33.781630 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:33.781890 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 17:06:33.781970 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 17:06:33.782477 28869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:33.777718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:41617 (local address 127.9.111.65:43109)
0806 17:06:33.778272 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:33.778286 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:33.778416 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:33.778735 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:33.778740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:33.778765 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:33.778865 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:33.778870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.779765 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.779768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:33.780724 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:33.780733 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.781469 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.781475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:33.781593 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:33.781618 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 17:06:33.782138 (+ 520us) client_negotiation.cc:240] Negotiation successful
0806 17:06:33.782327 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":92,"thread_start_us":174,"threads_started":1}
I20260806 17:06:33.782516 28868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:33.777717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:33005 (local address 127.9.111.65:33583)
0806 17:06:33.778111 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:33.778161 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:33.778315 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:33.778658 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:33.778661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:33.778683 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:33.778846 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:33.778852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.779765 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.779769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:33.780701 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:33.780711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.781606 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.781610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:33.781711 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:33.781734 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 17:06:33.782285 (+ 551us) client_negotiation.cc:240] Negotiation successful
0806 17:06:33.782361 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":40,"thread_start_us":217,"threads_started":1}
I20260806 17:06:33.783206 28795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4da25eef084b1a9ab58dad7c48688b" candidate_uuid: "2cb03c7c09434874955b88752a3b185e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1ea21308f84a6792d3cc4e280ae306" is_pre_election: true
I20260806 17:06:33.783206 28660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4da25eef084b1a9ab58dad7c48688b" candidate_uuid: "2cb03c7c09434874955b88752a3b185e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d615055377c4a95a1355a80e3894009" is_pre_election: true
I20260806 17:06:33.783546 28795 raft_consensus.cc:2454] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb03c7c09434874955b88752a3b185e in term 0.
I20260806 17:06:33.783552 28660 raft_consensus.cc:2454] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb03c7c09434874955b88752a3b185e in term 0.
I20260806 17:06:33.784093 28461 leader_election.cc:302] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d615055377c4a95a1355a80e3894009, 2cb03c7c09434874955b88752a3b185e, 5e1ea21308f84a6792d3cc4e280ae306; no voters:
I20260806 17:06:33.784382 28865 raft_consensus.cc:2781] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:06:33.784480 28865 raft_consensus.cc:482] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:06:33.784528 28865 raft_consensus.cc:3037] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:33.785633 28865 raft_consensus.cc:504] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1ea21308f84a6792d3cc4e280ae306" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41617 } } peers { permanent_uuid: "0d615055377c4a95a1355a80e3894009" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33005 } } peers { permanent_uuid: "2cb03c7c09434874955b88752a3b185e" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40027 } }
I20260806 17:06:33.786087 28865 leader_election.cc:288] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e [CANDIDATE]: Term 1 election: Requested vote from peers 5e1ea21308f84a6792d3cc4e280ae306 (127.9.111.67:41617), 0d615055377c4a95a1355a80e3894009 (127.9.111.66:33005)
I20260806 17:06:33.786465 28795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4da25eef084b1a9ab58dad7c48688b" candidate_uuid: "2cb03c7c09434874955b88752a3b185e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1ea21308f84a6792d3cc4e280ae306"
I20260806 17:06:33.786482 28660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4da25eef084b1a9ab58dad7c48688b" candidate_uuid: "2cb03c7c09434874955b88752a3b185e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d615055377c4a95a1355a80e3894009"
I20260806 17:06:33.786620 28795 raft_consensus.cc:3037] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:33.786630 28660 raft_consensus.cc:3037] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:33.787693 28795 raft_consensus.cc:2454] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb03c7c09434874955b88752a3b185e in term 1.
I20260806 17:06:33.787786 28660 raft_consensus.cc:2454] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb03c7c09434874955b88752a3b185e in term 1.
I20260806 17:06:33.788046 28461 leader_election.cc:302] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb03c7c09434874955b88752a3b185e, 5e1ea21308f84a6792d3cc4e280ae306; no voters:
I20260806 17:06:33.788304 28865 raft_consensus.cc:2781] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:33.788506 28865 raft_consensus.cc:686] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e [term 1 LEADER]: Becoming Leader. State: Replica: 2cb03c7c09434874955b88752a3b185e, State: Running, Role: LEADER
I20260806 17:06:33.788905 28865 consensus_queue.cc:237] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1ea21308f84a6792d3cc4e280ae306" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41617 } } peers { permanent_uuid: "0d615055377c4a95a1355a80e3894009" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33005 } } peers { permanent_uuid: "2cb03c7c09434874955b88752a3b185e" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40027 } }
I20260806 17:06:33.792413 28381 catalog_manager.cc:5495] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cb03c7c09434874955b88752a3b185e (127.9.111.65). New cstate: current_term: 1 leader_uuid: "2cb03c7c09434874955b88752a3b185e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1ea21308f84a6792d3cc4e280ae306" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d615055377c4a95a1355a80e3894009" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cb03c7c09434874955b88752a3b185e" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 40027 } health_report { overall_health: HEALTHY } } }
W20260806 17:06:33.794482 28573 tablet.cc:2369] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:06:33.849342 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:33.845489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58346 (local address 127.9.111.65:40027)
0806 17:06:33.845650 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:33.845655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:33.845678 (+ 23us) server_negotiation.cc:407] Connection header received
0806 17:06:33.845861 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:33.845864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:33.845944 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:33.846060 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:06:33.846328 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.846849 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.847747 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:33.847982 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:33.848126 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:33.848491 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:06:33.848894 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:06:33.849060 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:33.849154 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 17:06:33.849196 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 17:06:33.855578 28795 raft_consensus.cc:1261] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306 [term 1 FOLLOWER]: Refusing update from remote peer 2cb03c7c09434874955b88752a3b185e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:06:33.855578 28660 raft_consensus.cc:1261] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009 [term 1 FOLLOWER]: Refusing update from remote peer 2cb03c7c09434874955b88752a3b185e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:06:33.856420 28865 consensus_queue.cc:1035] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1ea21308f84a6792d3cc4e280ae306" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 41617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:06:33.856652 28870 consensus_queue.cc:1035] T 2f4da25eef084b1a9ab58dad7c48688b P 2cb03c7c09434874955b88752a3b185e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d615055377c4a95a1355a80e3894009" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 33005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:06:33.863116 28873 mvcc.cc:205] Tried to move back new op lower bound from 7315603430824030208 to 7315603430562160640. Current Snapshot: MvccSnapshot[applied={T|T < 7315603430824030208}]
I20260806 17:06:33.864943 28875 mvcc.cc:205] Tried to move back new op lower bound from 7315603430824030208 to 7315603430562160640. Current Snapshot: MvccSnapshot[applied={T|T < 7315603430824030208}]
I20260806 17:06:33.873000 28381 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:06:33.873186 28381 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:06:33.875809 28381 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4fef6c94ed949d79df815bd1908279b: Sending DeleteTablet for 3 replicas of tablet 2f4da25eef084b1a9ab58dad7c48688b
I20260806 17:06:33.876632 28774 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f4da25eef084b1a9ab58dad7c48688b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:06:33 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:39684
I20260806 17:06:33.876749 28640 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f4da25eef084b1a9ab58dad7c48688b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:06:33 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:51156
I20260806 17:06:33.877264 28884 tablet_replica.cc:330] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009: stopping tablet replica
I20260806 17:06:33.877372 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 28443
I20260806 17:06:33.877624 28884 raft_consensus.cc:2229] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:06:33.877983 28884 raft_consensus.cc:2258] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:06:33.879772 28884 ts_tablet_manager.cc:1905] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:06:33.882989 28884 ts_tablet_manager.cc:1918] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:06:33.883085 28883 tablet_replica.cc:330] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306: stopping tablet replica
I20260806 17:06:33.883126 28884 log.cc:1196] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/2f4da25eef084b1a9ab58dad7c48688b
I20260806 17:06:33.883422 28884 ts_tablet_manager.cc:1939] T 2f4da25eef084b1a9ab58dad7c48688b P 0d615055377c4a95a1355a80e3894009: Deleting consensus metadata
I20260806 17:06:33.883531 28883 raft_consensus.cc:2229] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:06:33.883898 28883 raft_consensus.cc:2258] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:06:33.884769 28368 catalog_manager.cc:4841] TS 0d615055377c4a95a1355a80e3894009 (127.9.111.66:33005): tablet 2f4da25eef084b1a9ab58dad7c48688b (table test-table [id=0fa9e0fdd79d40c1ab9939714453e0cc]) successfully deleted
I20260806 17:06:33.885594 28883 ts_tablet_manager.cc:1905] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 17:06:33.888751 28368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.111.65:40027 (error 108)
I20260806 17:06:33.889029 28883 ts_tablet_manager.cc:1918] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:06:33.889143 28883 log.cc:1196] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/2f4da25eef084b1a9ab58dad7c48688b
I20260806 17:06:33.889521 28883 ts_tablet_manager.cc:1939] T 2f4da25eef084b1a9ab58dad7c48688b P 5e1ea21308f84a6792d3cc4e280ae306: Deleting consensus metadata
I20260806 17:06:33.889711 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 28577
I20260806 17:06:33.890038 28853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:33.889693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:40027 (local address 127.0.0.1:58350)
0806 17:06:33.889854 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:33.889936 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.65:40027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 17:06:33.890447 28368 catalog_manager.cc:4584] TS 2cb03c7c09434874955b88752a3b185e (127.9.111.65:40027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f4da25eef084b1a9ab58dad7c48688b: Network error: Client connection negotiation failed: client connection to 127.9.111.65:40027: connect: Connection refused (error 111)
I20260806 17:06:33.890944 28367 catalog_manager.cc:4841] TS 5e1ea21308f84a6792d3cc4e280ae306 (127.9.111.67:41617): tablet 2f4da25eef084b1a9ab58dad7c48688b (table test-table [id=0fa9e0fdd79d40c1ab9939714453e0cc]) successfully deleted
I20260806 17:06:33.896561 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 28711
I20260806 17:06:33.903155 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 28349
2026-08-06T17:06:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2561 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T17:06:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:33Z Disabled control of system clock
I20260806 17:06:33.937930 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:40269
--webserver_interface=127.9.111.126
--webserver_port=0
--builtin_ntp_servers=127.9.111.84:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:40269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 17:06:34.053958 28889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:34.054237 28889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:34.054275 28889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:34.058444 28889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:34.058518 28889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:34.058537 28889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:34.058553 28889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:34.058569 28889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:34.067054 28889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:40269
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:40269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:34.068729 28889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:34.070528 28889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:34.079922 28897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:34.079906 28894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:34.079910 28895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:34.080364 28889 server_base.cc:1034] running on GCE node
I20260806 17:06:34.080979 28889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:34.082261 28889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:34.083453 28889 hybrid_clock.cc:648] HybridClock initialized: now 1786035994083434 us; error 34 us; skew 500 ppm
I20260806 17:06:34.085520 28889 webserver.cc:460] Webserver started at http://127.9.111.126:43109/ using document root <none> and password file <none>
I20260806 17:06:34.086179 28889 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:34.086247 28889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:34.086565 28889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:34.088985 28889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "a452c469da9f4687bd3fa48de8c7c6d0"
format_stamp: "Formatted at 2026-08-06 17:06:34 on dist-test-slave-69rg"
server_key: "0bae8f36479a1ed2749c8096b7909252"
server_key_iv: "8604ee47c3335355228f4bd5761df96c"
server_key_version: "encryptionkey@0"
I20260806 17:06:34.089635 28889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "a452c469da9f4687bd3fa48de8c7c6d0"
format_stamp: "Formatted at 2026-08-06 17:06:34 on dist-test-slave-69rg"
server_key: "0bae8f36479a1ed2749c8096b7909252"
server_key_iv: "8604ee47c3335355228f4bd5761df96c"
server_key_version: "encryptionkey@0"
I20260806 17:06:34.093664 28889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 17:06:34.096469 28903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:34.097908 28889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:06:34.098059 28889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "a452c469da9f4687bd3fa48de8c7c6d0"
format_stamp: "Formatted at 2026-08-06 17:06:34 on dist-test-slave-69rg"
server_key: "0bae8f36479a1ed2749c8096b7909252"
server_key_iv: "8604ee47c3335355228f4bd5761df96c"
server_key_version: "encryptionkey@0"
I20260806 17:06:34.098191 28889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:34.110278 28889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:34.111009 28889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:34.111215 28889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:34.120788 28889 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:40269
I20260806 17:06:34.120810 28957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:40269 every 8 connection(s)
I20260806 17:06:34.122138 28889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:34.125195 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 28889
I20260806 17:06:34.125351 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:34.125649 9661 external_mini_cluster.cc:1428] Setting key 2184a51c6db034f85eb6aabc9dbab878
I20260806 17:06:34.126022 28958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:34.134159 28958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0: Bootstrap starting.
I20260806 17:06:34.136698 28958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:34.137557 28958 log.cc:824] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:34.140065 28958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0: No bootstrap required, opened a new log
I20260806 17:06:34.141882 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:34.127189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.9.111.126:40269)
0806 17:06:34.127999 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:34.128021 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:34.128068 (+ 47us) server_negotiation.cc:407] Connection header received
0806 17:06:34.129152 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:34.129178 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:34.129768 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:34.130212 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:34.130706 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.132282 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.133185 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.133951 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.134119 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:34.139594 (+ 5475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:34.139601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:34.139619 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:34.140167 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:34.140249 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:34.140278 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:34.140387 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:34.140551 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 17:06:34.140852 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":130,"threads_started":1}
I20260806 17:06:34.144894 28958 raft_consensus.cc:348] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a452c469da9f4687bd3fa48de8c7c6d0" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 40269 } }
I20260806 17:06:34.145119 28958 raft_consensus.cc:374] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:34.145165 28958 raft_consensus.cc:729] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a452c469da9f4687bd3fa48de8c7c6d0, State: Initialized, Role: FOLLOWER
I20260806 17:06:34.145592 28958 consensus_queue.cc:260] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a452c469da9f4687bd3fa48de8c7c6d0" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 40269 } }
I20260806 17:06:34.145731 28958 raft_consensus.cc:388] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:34.145781 28958 raft_consensus.cc:482] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:34.145920 28958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:34.146868 28958 raft_consensus.cc:504] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a452c469da9f4687bd3fa48de8c7c6d0" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 40269 } }
I20260806 17:06:34.147189 28958 leader_election.cc:302] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a452c469da9f4687bd3fa48de8c7c6d0; no voters:
I20260806 17:06:34.147469 28958 leader_election.cc:288] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:34.147691 28963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:34.147958 28963 raft_consensus.cc:686] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0 [term 1 LEADER]: Becoming Leader. State: Replica: a452c469da9f4687bd3fa48de8c7c6d0, State: Running, Role: LEADER
I20260806 17:06:34.148276 28963 consensus_queue.cc:237] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a452c469da9f4687bd3fa48de8c7c6d0" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 40269 } }
I20260806 17:06:34.148908 28958 sys_catalog.cc:564] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:34.150468 28965 sys_catalog.cc:455] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a452c469da9f4687bd3fa48de8c7c6d0. Latest consensus state: current_term: 1 leader_uuid: "a452c469da9f4687bd3fa48de8c7c6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a452c469da9f4687bd3fa48de8c7c6d0" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 40269 } } }
I20260806 17:06:34.150605 28965 sys_catalog.cc:458] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:34.149979 28964 sys_catalog.cc:455] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a452c469da9f4687bd3fa48de8c7c6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a452c469da9f4687bd3fa48de8c7c6d0" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 40269 } } }
I20260806 17:06:34.150962 28964 sys_catalog.cc:458] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0 [sys.catalog]: This master's current role is: LEADER
W20260806 17:06:34.154292 28979 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:06:34.154379 28979 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:06:34.154449 28973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:34.155236 28973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:34.160881 28973 catalog_manager.cc:1347] Generated new cluster ID: 6f038ee2c31e46a78f802b42614fe4d6
I20260806 17:06:34.160983 28973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:34.175967 28973 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:34.177034 28973 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:34.183808 28973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0: Generated new TSK 0
I20260806 17:06:34.184607 28973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:06:34.188712 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:40269
--builtin_ntp_servers=127.9.111.84:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 17:06:34.317585 28982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:34.318058 28982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:34.318161 28982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:34.323047 28982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:34.323132 28982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:34.323225 28982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:06:34.328938 28982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:40269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:34.330283 28982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:34.331737 28982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:34.341622 28987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:34.341662 28990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:34.341740 28982 server_base.cc:1034] running on GCE node
W20260806 17:06:34.341616 28988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:34.342289 28982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:34.342989 28982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:34.344180 28982 hybrid_clock.cc:648] HybridClock initialized: now 1786035994344163 us; error 30 us; skew 500 ppm
I20260806 17:06:34.346063 28982 webserver.cc:460] Webserver started at http://127.9.111.65:41889/ using document root <none> and password file <none>
I20260806 17:06:34.346647 28982 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:34.346743 28982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:34.346980 28982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:34.349020 28982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "8cfb62e1db65439b9045f0af742caa83"
format_stamp: "Formatted at 2026-08-06 17:06:34 on dist-test-slave-69rg"
server_key: "3ee9c42f53d445c327d01968b3d52a3c"
server_key_iv: "270a4a5da7438ca846401b26d08a8ece"
server_key_version: "encryptionkey@0"
I20260806 17:06:34.349858 28982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "8cfb62e1db65439b9045f0af742caa83"
format_stamp: "Formatted at 2026-08-06 17:06:34 on dist-test-slave-69rg"
server_key: "3ee9c42f53d445c327d01968b3d52a3c"
server_key_iv: "270a4a5da7438ca846401b26d08a8ece"
server_key_version: "encryptionkey@0"
I20260806 17:06:34.353631 28982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 17:06:34.356268 28996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:34.357522 28982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 17:06:34.357662 28982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "8cfb62e1db65439b9045f0af742caa83"
format_stamp: "Formatted at 2026-08-06 17:06:34 on dist-test-slave-69rg"
server_key: "3ee9c42f53d445c327d01968b3d52a3c"
server_key_iv: "270a4a5da7438ca846401b26d08a8ece"
server_key_version: "encryptionkey@0"
I20260806 17:06:34.357785 28982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:34.374397 28982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:34.375094 28982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:34.375355 28982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:34.375957 28982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:34.376893 28982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:34.376966 28982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:34.377033 28982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:34.377086 28982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:34.386021 28982 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:39719
I20260806 17:06:34.386049 29111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:39719 every 8 connection(s)
I20260806 17:06:34.386931 28982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:06:34.396040 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:34.388785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:51337 (local address 127.9.111.126:40269)
0806 17:06:34.388960 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:34.388966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:34.390060 (+ 1094us) server_negotiation.cc:407] Connection header received
0806 17:06:34.391421 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:34.391424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:34.391471 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:34.391548 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:34.392435 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.392979 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.394125 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.394344 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.394429 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:34.395585 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:34.395588 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:34.395591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:34.395604 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:34.395632 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:34.395640 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:34.395715 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:34.395845 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 17:06:34.395914 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 17:06:34.396883 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 28982
I20260806 17:06:34.397027 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:06:34.397120 29114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:34.389200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:40269 (local address 127.9.111.65:51337)
0806 17:06:34.389902 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:34.389948 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:34.391204 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:34.391710 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:34.391716 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:34.391798 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:34.392309 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:34.392322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.393085 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.393087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:34.394013 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:34.394023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.394485 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.394488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:34.394631 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:34.395317 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:06:34.395318 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:34.395358 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:06:34.395421 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:34.395722 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:34.395729 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:34.395733 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 17:06:34.395826 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 17:06:34.396042 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":119,"threads_started":1}
I20260806 17:06:34.397418 9661 external_mini_cluster.cc:1428] Setting key 14c3ee0579fe6fe90dfa334299ff0016
I20260806 17:06:34.398492 29112 heartbeater.cc:347] Connected to a master server at 127.9.111.126:40269
I20260806 17:06:34.398769 29112 heartbeater.cc:464] Registering TS with master...
I20260806 17:06:34.399317 29112 heartbeater.cc:511] Master 127.9.111.126:40269 requested a full tablet report, sending...
I20260806 17:06:34.399473 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:40269
--builtin_ntp_servers=127.9.111.84:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 17:06:34.401078 28921 ts_manager.cc:194] Registered new tserver with Master: 8cfb62e1db65439b9045f0af742caa83 (127.9.111.65:39719)
I20260806 17:06:34.402413 28921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.111.65:51337
W20260806 17:06:34.516404 29115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:34.516723 29115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:34.516800 29115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:34.520998 29115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:34.521121 29115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:34.521219 29115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:06:34.526098 29115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:40269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:34.527245 29115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:34.528679 29115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:34.538512 29121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:34.538527 29120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:34.538794 29115 server_base.cc:1034] running on GCE node
W20260806 17:06:34.538528 29123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:34.539345 29115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:34.539996 29115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:34.541225 29115 hybrid_clock.cc:648] HybridClock initialized: now 1786035994541213 us; error 31 us; skew 500 ppm
I20260806 17:06:34.547488 29115 webserver.cc:460] Webserver started at http://127.9.111.66:45357/ using document root <none> and password file <none>
I20260806 17:06:34.548211 29115 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:34.548326 29115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:34.548609 29115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:34.551287 29115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "533ccda3bae04fb3969458d59827aa9b"
format_stamp: "Formatted at 2026-08-06 17:06:34 on dist-test-slave-69rg"
server_key: "b6a92f9a61032bcc9f0b07b414c8bbed"
server_key_iv: "3d74d218c89823f4c7c75fed0554d6c4"
server_key_version: "encryptionkey@0"
I20260806 17:06:34.551934 29115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "533ccda3bae04fb3969458d59827aa9b"
format_stamp: "Formatted at 2026-08-06 17:06:34 on dist-test-slave-69rg"
server_key: "b6a92f9a61032bcc9f0b07b414c8bbed"
server_key_iv: "3d74d218c89823f4c7c75fed0554d6c4"
server_key_version: "encryptionkey@0"
I20260806 17:06:34.555982 29115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 17:06:34.558597 29129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:34.559926 29115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:06:34.560087 29115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "533ccda3bae04fb3969458d59827aa9b"
format_stamp: "Formatted at 2026-08-06 17:06:34 on dist-test-slave-69rg"
server_key: "b6a92f9a61032bcc9f0b07b414c8bbed"
server_key_iv: "3d74d218c89823f4c7c75fed0554d6c4"
server_key_version: "encryptionkey@0"
I20260806 17:06:34.560214 29115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:34.582121 29115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:34.582876 29115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:34.583089 29115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:34.583752 29115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:34.584751 29115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:34.584826 29115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:34.584900 29115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:34.584942 29115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:34.593883 29115 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:34547
I20260806 17:06:34.593932 29244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:34547 every 8 connection(s)
I20260806 17:06:34.594811 29115 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:06:34.598310 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 29115
I20260806 17:06:34.598464 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:06:34.598766 9661 external_mini_cluster.cc:1428] Setting key 9c8305b04b2901e6b5212d9e3ee291c7
I20260806 17:06:34.600834 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:40269
--builtin_ntp_servers=127.9.111.84:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 17:06:34.605226 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:34.596764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:42121 (local address 127.9.111.126:40269)
0806 17:06:34.596929 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:34.596934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:34.598176 (+ 1242us) server_negotiation.cc:407] Connection header received
0806 17:06:34.600114 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:34.600117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:34.600194 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:34.600336 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:34.601330 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.601938 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.603138 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.603384 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.603501 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:34.604753 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:34.604756 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:34.604758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:34.604771 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:34.604792 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:34.604800 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:34.604868 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:34.605052 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 17:06:34.605105 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 17:06:34.606531 29247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:34.597226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:40269 (local address 127.9.111.66:42121)
0806 17:06:34.597991 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:34.598051 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:34.599874 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:34.600528 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:34.600535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:34.600619 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:34.601182 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:34.601198 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.602068 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.602077 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:34.603016 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:34.603026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.603543 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.603547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:34.603717 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:34.604421 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:06:34.604425 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:34.604482 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:06:34.604569 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:34.604892 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:34.604904 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:34.604913 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 17:06:34.605047 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 17:06:34.605362 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":126,"threads_started":1}
I20260806 17:06:34.607807 29245 heartbeater.cc:347] Connected to a master server at 127.9.111.126:40269
I20260806 17:06:34.608073 29245 heartbeater.cc:464] Registering TS with master...
I20260806 17:06:34.608633 29245 heartbeater.cc:511] Master 127.9.111.126:40269 requested a full tablet report, sending...
I20260806 17:06:34.609690 28921 ts_manager.cc:194] Registered new tserver with Master: 533ccda3bae04fb3969458d59827aa9b (127.9.111.66:34547)
I20260806 17:06:34.610296 28921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.111.66:42121
W20260806 17:06:34.717517 29248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:34.717871 29248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:34.717949 29248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:34.722007 29248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:34.722116 29248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:34.722218 29248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:06:34.727166 29248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:40269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:34.728430 29248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:34.729918 29248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:34.740196 29256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:34.740177 29253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:34.740163 29254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:34.740438 29248 server_base.cc:1034] running on GCE node
I20260806 17:06:34.740933 29248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:34.741647 29248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:34.742863 29248 hybrid_clock.cc:648] HybridClock initialized: now 1786035994742841 us; error 35 us; skew 500 ppm
I20260806 17:06:34.744787 29248 webserver.cc:460] Webserver started at http://127.9.111.67:41339/ using document root <none> and password file <none>
I20260806 17:06:34.745388 29248 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:34.745486 29248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:34.745735 29248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:34.747915 29248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "c2ffb33985044eb9bf409474f946e289"
format_stamp: "Formatted at 2026-08-06 17:06:34 on dist-test-slave-69rg"
server_key: "bdbccd9bc71147e77d62c4008e59e1d1"
server_key_iv: "d17f0e1a85a19fdc43811b624605eae3"
server_key_version: "encryptionkey@0"
I20260806 17:06:34.748574 29248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "c2ffb33985044eb9bf409474f946e289"
format_stamp: "Formatted at 2026-08-06 17:06:34 on dist-test-slave-69rg"
server_key: "bdbccd9bc71147e77d62c4008e59e1d1"
server_key_iv: "d17f0e1a85a19fdc43811b624605eae3"
server_key_version: "encryptionkey@0"
I20260806 17:06:34.752748 29248 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 17:06:34.755312 29262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:34.756531 29248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 17:06:34.756676 29248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "c2ffb33985044eb9bf409474f946e289"
format_stamp: "Formatted at 2026-08-06 17:06:34 on dist-test-slave-69rg"
server_key: "bdbccd9bc71147e77d62c4008e59e1d1"
server_key_iv: "d17f0e1a85a19fdc43811b624605eae3"
server_key_version: "encryptionkey@0"
I20260806 17:06:34.756810 29248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:34.771766 29248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:34.772496 29248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:34.772719 29248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:34.773334 29248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:34.774376 29248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:34.774449 29248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:34.774523 29248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:34.774566 29248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:34.783638 29248 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:40409
I20260806 17:06:34.783706 29377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:40409 every 8 connection(s)
I20260806 17:06:34.784600 29248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:06:34.789312 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 29248
I20260806 17:06:34.789472 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:06:34.789809 9661 external_mini_cluster.cc:1428] Setting key 9796e7b1ed3b6dcd5748ee2aa473cbfb
I20260806 17:06:34.794687 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:34.786525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:46715 (local address 127.9.111.126:40269)
0806 17:06:34.786745 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:34.786749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:34.787793 (+ 1044us) server_negotiation.cc:407] Connection header received
0806 17:06:34.789497 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:34.789502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:34.789569 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:34.789655 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:34.790885 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.791628 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.792954 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.793166 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.793255 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:34.794291 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:34.794294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:34.794296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:34.794308 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:34.794326 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:34.794333 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:34.794391 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:34.794528 (+ 137us) server_negotiation.cc:299] Negotiation successful
0806 17:06:34.794575 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260806 17:06:34.795512 29380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:34.786917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:40269 (local address 127.9.111.67:46715)
0806 17:06:34.787596 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:34.787670 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:34.789211 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:34.789890 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:34.789900 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:34.790013 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:34.790751 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:34.790774 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.791786 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.791789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:34.792838 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:34.792848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.793304 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.793306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:34.793444 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:34.794052 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:06:34.794053 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:34.794092 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:06:34.794155 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:34.794406 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:34.794412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:34.794416 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 17:06:34.794510 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 17:06:34.794723 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":103,"threads_started":1}
I20260806 17:06:34.796586 29378 heartbeater.cc:347] Connected to a master server at 127.9.111.126:40269
I20260806 17:06:34.796828 29378 heartbeater.cc:464] Registering TS with master...
I20260806 17:06:34.797351 29378 heartbeater.cc:511] Master 127.9.111.126:40269 requested a full tablet report, sending...
I20260806 17:06:34.798467 28921 ts_manager.cc:194] Registered new tserver with Master: c2ffb33985044eb9bf409474f946e289 (127.9.111.67:40409)
I20260806 17:06:34.798993 28921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.111.67:46715
I20260806 17:06:34.803695 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 17:06:34.809187 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:34.805638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37826 (local address 127.9.111.126:40269)
0806 17:06:34.805774 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:34.805777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:34.805979 (+ 202us) server_negotiation.cc:407] Connection header received
0806 17:06:34.806071 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:34.806073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:34.806110 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:34.806177 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:34.806461 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.806959 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.807766 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.807972 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.808047 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:34.808819 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:34.808822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:34.808824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:34.808836 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:34.808854 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:34.808861 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:34.808959 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:34.809055 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 17:06:34.809084 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 17:06:34.812043 28921 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:06:34.815145 28921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:06:34.829496 29391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:34.823733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52364 (local address 127.9.111.66:34547)
0806 17:06:34.824195 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:34.824208 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:34.824843 (+ 635us) server_negotiation.cc:407] Connection header received
0806 17:06:34.825019 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:34.825024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:34.825239 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:34.825384 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:34.825908 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.826679 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.828217 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.828800 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.828927 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:34.829123 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:34.829239 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 17:06:34.829333 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":134,"threads_started":1}
I20260806 17:06:34.829495 29390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:34.823587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:34547 (local address 127.0.0.1:52364)
0806 17:06:34.824459 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:34.824514 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:34.824648 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:34.825529 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:34.825532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:34.825572 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:34.825741 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:34.825750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.826802 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.826808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:34.828066 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:34.828078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.828929 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.828933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:34.829039 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:34.829076 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 17:06:34.829203 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 17:06:34.829325 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":2,"spinlock_wait_cycles":6528,"thread_start_us":77,"threads_started":1}
I20260806 17:06:34.830197 29389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:34.823916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:39719 (local address 127.0.0.1:35010)
0806 17:06:34.825076 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:34.825088 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:34.825163 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:34.825793 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:34.825798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:34.825844 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:34.825953 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:34.825960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.827346 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.827350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:34.828467 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:34.828476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.829771 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.829774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:34.829909 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:34.829933 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 17:06:34.830002 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 17:06:34.830064 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"thread_start_us":86,"threads_started":1}
I20260806 17:06:34.830709 29393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:34.824365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.9.111.65:39719)
0806 17:06:34.824856 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:34.824867 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:34.825263 (+ 396us) server_negotiation.cc:407] Connection header received
0806 17:06:34.825347 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:34.825351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:34.825511 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:34.825649 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:34.826075 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.827234 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.828596 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.829397 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.829523 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:34.829702 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:34.830447 (+ 745us) server_negotiation.cc:299] Negotiation successful
0806 17:06:34.830548 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":103,"threads_started":1}
I20260806 17:06:34.831146 29392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:34.823607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:40409 (local address 127.0.0.1:35124)
0806 17:06:34.824459 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:34.824513 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:34.824648 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:34.825524 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:34.825529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:34.825552 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:34.825739 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:34.825746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.827180 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.827183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:34.829432 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:34.829445 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.830725 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.830732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:34.830831 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:34.830854 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 17:06:34.830928 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 17:06:34.831007 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":1174,"spinlock_wait_cycles":21760,"thread_start_us":144,"threads_started":1}
I20260806 17:06:34.831875 29394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:34.824333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35124 (local address 127.9.111.67:40409)
0806 17:06:34.825049 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:34.825059 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:34.825082 (+ 23us) server_negotiation.cc:407] Connection header received
0806 17:06:34.825161 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:34.825169 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:34.825361 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:34.825517 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:34.825873 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.827056 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.829570 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.830391 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.830488 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:34.830645 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:34.831603 (+ 958us) server_negotiation.cc:299] Negotiation successful
0806 17:06:34.831699 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":149,"threads_started":1}
I20260806 17:06:34.833020 29177 tablet_service.cc:1467] Processing CreateTablet for tablet 274562ff4e3d4b54aada1a5b55d278b0 (DEFAULT_TABLE table=test-table [id=dbd9e8ba26b1402f87d9a4bfaa1e25f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:06:34.833554 29044 tablet_service.cc:1467] Processing CreateTablet for tablet 274562ff4e3d4b54aada1a5b55d278b0 (DEFAULT_TABLE table=test-table [id=dbd9e8ba26b1402f87d9a4bfaa1e25f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:06:34.834215 29177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274562ff4e3d4b54aada1a5b55d278b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:34.833956 29310 tablet_service.cc:1467] Processing CreateTablet for tablet 274562ff4e3d4b54aada1a5b55d278b0 (DEFAULT_TABLE table=test-table [id=dbd9e8ba26b1402f87d9a4bfaa1e25f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:06:34.834606 29044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274562ff4e3d4b54aada1a5b55d278b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:34.834944 29310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274562ff4e3d4b54aada1a5b55d278b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:34.841029 29395 tablet_bootstrap.cc:492] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289: Bootstrap starting.
I20260806 17:06:34.842474 29397 tablet_bootstrap.cc:492] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b: Bootstrap starting.
I20260806 17:06:34.843426 29395 tablet_bootstrap.cc:654] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:34.844177 29397 tablet_bootstrap.cc:654] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:34.844175 29395 log.cc:824] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:34.844861 29396 tablet_bootstrap.cc:492] T 274562ff4e3d4b54aada1a5b55d278b0 P 8cfb62e1db65439b9045f0af742caa83: Bootstrap starting.
I20260806 17:06:34.844988 29397 log.cc:824] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:34.846446 29395 tablet_bootstrap.cc:492] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289: No bootstrap required, opened a new log
I20260806 17:06:34.846640 29395 ts_tablet_manager.cc:1397] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 17:06:34.847091 29396 tablet_bootstrap.cc:654] T 274562ff4e3d4b54aada1a5b55d278b0 P 8cfb62e1db65439b9045f0af742caa83: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:34.847443 29397 tablet_bootstrap.cc:492] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b: No bootstrap required, opened a new log
I20260806 17:06:34.847626 29397 ts_tablet_manager.cc:1397] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 17:06:34.847898 29396 log.cc:824] T 274562ff4e3d4b54aada1a5b55d278b0 P 8cfb62e1db65439b9045f0af742caa83: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:34.849673 29396 tablet_bootstrap.cc:492] T 274562ff4e3d4b54aada1a5b55d278b0 P 8cfb62e1db65439b9045f0af742caa83: No bootstrap required, opened a new log
I20260806 17:06:34.849905 29396 ts_tablet_manager.cc:1397] T 274562ff4e3d4b54aada1a5b55d278b0 P 8cfb62e1db65439b9045f0af742caa83: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 17:06:34.851258 29395 raft_consensus.cc:348] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533ccda3bae04fb3969458d59827aa9b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34547 } } peers { permanent_uuid: "c2ffb33985044eb9bf409474f946e289" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 40409 } } peers { permanent_uuid: "8cfb62e1db65439b9045f0af742caa83" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39719 } }
I20260806 17:06:34.851536 29395 raft_consensus.cc:374] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:34.851596 29395 raft_consensus.cc:729] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ffb33985044eb9bf409474f946e289, State: Initialized, Role: FOLLOWER
I20260806 17:06:34.851817 29397 raft_consensus.cc:348] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533ccda3bae04fb3969458d59827aa9b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34547 } } peers { permanent_uuid: "c2ffb33985044eb9bf409474f946e289" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 40409 } } peers { permanent_uuid: "8cfb62e1db65439b9045f0af742caa83" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39719 } }
I20260806 17:06:34.852046 29397 raft_consensus.cc:374] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:34.852100 29397 raft_consensus.cc:729] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533ccda3bae04fb3969458d59827aa9b, State: Initialized, Role: FOLLOWER
I20260806 17:06:34.852316 29395 consensus_queue.cc:260] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533ccda3bae04fb3969458d59827aa9b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34547 } } peers { permanent_uuid: "c2ffb33985044eb9bf409474f946e289" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 40409 } } peers { permanent_uuid: "8cfb62e1db65439b9045f0af742caa83" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39719 } }
I20260806 17:06:34.852537 29397 consensus_queue.cc:260] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533ccda3bae04fb3969458d59827aa9b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34547 } } peers { permanent_uuid: "c2ffb33985044eb9bf409474f946e289" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 40409 } } peers { permanent_uuid: "8cfb62e1db65439b9045f0af742caa83" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39719 } }
I20260806 17:06:34.853112 29378 heartbeater.cc:503] Master 127.9.111.126:40269 was elected leader, sending a full tablet report...
I20260806 17:06:34.853410 29395 ts_tablet_manager.cc:1428] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 17:06:34.854058 29396 raft_consensus.cc:348] T 274562ff4e3d4b54aada1a5b55d278b0 P 8cfb62e1db65439b9045f0af742caa83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533ccda3bae04fb3969458d59827aa9b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34547 } } peers { permanent_uuid: "c2ffb33985044eb9bf409474f946e289" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 40409 } } peers { permanent_uuid: "8cfb62e1db65439b9045f0af742caa83" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39719 } }
I20260806 17:06:34.854329 29396 raft_consensus.cc:374] T 274562ff4e3d4b54aada1a5b55d278b0 P 8cfb62e1db65439b9045f0af742caa83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:34.854400 29396 raft_consensus.cc:729] T 274562ff4e3d4b54aada1a5b55d278b0 P 8cfb62e1db65439b9045f0af742caa83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cfb62e1db65439b9045f0af742caa83, State: Initialized, Role: FOLLOWER
I20260806 17:06:34.854838 29396 consensus_queue.cc:260] T 274562ff4e3d4b54aada1a5b55d278b0 P 8cfb62e1db65439b9045f0af742caa83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533ccda3bae04fb3969458d59827aa9b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34547 } } peers { permanent_uuid: "c2ffb33985044eb9bf409474f946e289" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 40409 } } peers { permanent_uuid: "8cfb62e1db65439b9045f0af742caa83" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39719 } }
I20260806 17:06:34.855480 29112 heartbeater.cc:503] Master 127.9.111.126:40269 was elected leader, sending a full tablet report...
I20260806 17:06:34.856284 29397 ts_tablet_manager.cc:1428] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260806 17:06:34.856737 29396 ts_tablet_manager.cc:1428] T 274562ff4e3d4b54aada1a5b55d278b0 P 8cfb62e1db65439b9045f0af742caa83: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 17:06:34.856959 29245 heartbeater.cc:503] Master 127.9.111.126:40269 was elected leader, sending a full tablet report...
W20260806 17:06:34.888108 29113 tablet.cc:2369] T 274562ff4e3d4b54aada1a5b55d278b0 P 8cfb62e1db65439b9045f0af742caa83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:06:34.931336 29402 raft_consensus.cc:482] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:06:34.931568 29402 raft_consensus.cc:504] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533ccda3bae04fb3969458d59827aa9b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34547 } } peers { permanent_uuid: "c2ffb33985044eb9bf409474f946e289" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 40409 } } peers { permanent_uuid: "8cfb62e1db65439b9045f0af742caa83" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39719 } }
I20260806 17:06:34.932775 29402 leader_election.cc:288] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ffb33985044eb9bf409474f946e289 (127.9.111.67:40409), 8cfb62e1db65439b9045f0af742caa83 (127.9.111.65:39719)
I20260806 17:06:34.937899 29247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:34.933035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:40409 (local address 127.9.111.66:59541)
0806 17:06:34.933171 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:34.933188 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:34.933322 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:34.933635 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:34.933641 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:34.933665 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:34.933840 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:34.933850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.934994 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.934996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:34.936281 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:34.936292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.937325 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.937329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:34.937494 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:34.937543 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 17:06:34.937617 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 17:06:34.937711 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":64}
I20260806 17:06:34.937918 29393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:34.933179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:44939 (local address 127.9.111.65:39719)
0806 17:06:34.933322 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:34.933327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:34.933750 (+ 423us) server_negotiation.cc:407] Connection header received
0806 17:06:34.933807 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:34.933809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:34.933873 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:34.933999 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:34.934252 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.935043 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.936480 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.937168 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.937305 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:34.937361 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:34.937682 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 17:06:34.937763 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 17:06:34.937918 29394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:34.933154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:59541 (local address 127.9.111.67:40409)
0806 17:06:34.933302 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:34.933307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:34.933328 (+ 21us) server_negotiation.cc:407] Connection header received
0806 17:06:34.933448 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:34.933451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:34.933508 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:34.933608 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:34.933983 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.934851 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.936463 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.937161 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.937305 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:34.937361 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:34.937679 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 17:06:34.937766 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 17:06:34.937906 29404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:34.933045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:39719 (local address 127.9.111.66:44939)
0806 17:06:34.933631 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:34.933644 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:34.933732 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:34.934013 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:34.934015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:34.934031 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:34.934131 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:34.934135 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.935157 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.935160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:34.936347 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:34.936356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.937325 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.937329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:34.937500 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:34.937536 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 17:06:34.937617 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 17:06:34.937710 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":253,"thread_start_us":160,"threads_started":1}
I20260806 17:06:34.938673 29331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274562ff4e3d4b54aada1a5b55d278b0" candidate_uuid: "533ccda3bae04fb3969458d59827aa9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ffb33985044eb9bf409474f946e289" is_pre_election: true
I20260806 17:06:34.938761 29064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274562ff4e3d4b54aada1a5b55d278b0" candidate_uuid: "533ccda3bae04fb3969458d59827aa9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cfb62e1db65439b9045f0af742caa83" is_pre_election: true
I20260806 17:06:34.939023 29331 raft_consensus.cc:2454] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533ccda3bae04fb3969458d59827aa9b in term 0.
I20260806 17:06:34.939072 29064 raft_consensus.cc:2454] T 274562ff4e3d4b54aada1a5b55d278b0 P 8cfb62e1db65439b9045f0af742caa83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533ccda3bae04fb3969458d59827aa9b in term 0.
I20260806 17:06:34.939558 29133 leader_election.cc:302] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533ccda3bae04fb3969458d59827aa9b, 8cfb62e1db65439b9045f0af742caa83; no voters:
I20260806 17:06:34.939857 29402 raft_consensus.cc:2781] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:06:34.939965 29402 raft_consensus.cc:482] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:06:34.940013 29402 raft_consensus.cc:3037] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:34.941053 29402 raft_consensus.cc:504] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533ccda3bae04fb3969458d59827aa9b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34547 } } peers { permanent_uuid: "c2ffb33985044eb9bf409474f946e289" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 40409 } } peers { permanent_uuid: "8cfb62e1db65439b9045f0af742caa83" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39719 } }
I20260806 17:06:34.941458 29402 leader_election.cc:288] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [CANDIDATE]: Term 1 election: Requested vote from peers c2ffb33985044eb9bf409474f946e289 (127.9.111.67:40409), 8cfb62e1db65439b9045f0af742caa83 (127.9.111.65:39719)
I20260806 17:06:34.941857 29331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274562ff4e3d4b54aada1a5b55d278b0" candidate_uuid: "533ccda3bae04fb3969458d59827aa9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ffb33985044eb9bf409474f946e289"
I20260806 17:06:34.941824 29064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274562ff4e3d4b54aada1a5b55d278b0" candidate_uuid: "533ccda3bae04fb3969458d59827aa9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cfb62e1db65439b9045f0af742caa83"
I20260806 17:06:34.941977 29331 raft_consensus.cc:3037] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:34.941977 29064 raft_consensus.cc:3037] T 274562ff4e3d4b54aada1a5b55d278b0 P 8cfb62e1db65439b9045f0af742caa83 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:34.943233 29331 raft_consensus.cc:2454] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533ccda3bae04fb3969458d59827aa9b in term 1.
I20260806 17:06:34.943233 29064 raft_consensus.cc:2454] T 274562ff4e3d4b54aada1a5b55d278b0 P 8cfb62e1db65439b9045f0af742caa83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533ccda3bae04fb3969458d59827aa9b in term 1.
I20260806 17:06:34.943558 29133 leader_election.cc:302] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533ccda3bae04fb3969458d59827aa9b, 8cfb62e1db65439b9045f0af742caa83; no voters:
I20260806 17:06:34.943742 29402 raft_consensus.cc:2781] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:34.943948 29402 raft_consensus.cc:686] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [term 1 LEADER]: Becoming Leader. State: Replica: 533ccda3bae04fb3969458d59827aa9b, State: Running, Role: LEADER
I20260806 17:06:34.944280 29402 consensus_queue.cc:237] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533ccda3bae04fb3969458d59827aa9b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34547 } } peers { permanent_uuid: "c2ffb33985044eb9bf409474f946e289" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 40409 } } peers { permanent_uuid: "8cfb62e1db65439b9045f0af742caa83" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39719 } }
I20260806 17:06:34.947284 28921 catalog_manager.cc:5495] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b reported cstate change: term changed from 0 to 1, leader changed from <none> to 533ccda3bae04fb3969458d59827aa9b (127.9.111.66). New cstate: current_term: 1 leader_uuid: "533ccda3bae04fb3969458d59827aa9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533ccda3bae04fb3969458d59827aa9b" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ffb33985044eb9bf409474f946e289" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 40409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cfb62e1db65439b9045f0af742caa83" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39719 } health_report { overall_health: UNKNOWN } } }
I20260806 17:06:34.984719 29391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:34.981275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52374 (local address 127.9.111.66:34547)
0806 17:06:34.981401 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:34.981404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:34.981416 (+ 12us) server_negotiation.cc:407] Connection header received
0806 17:06:34.981484 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:34.981486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:34.981540 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:34.981633 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:06:34.981950 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.982455 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.983289 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:34.983486 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:34.983558 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:34.983871 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:06:34.984257 (+ 386us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:06:34.984365 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:34.984548 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 17:06:34.984581 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 17:06:34.990445 29331 raft_consensus.cc:1261] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289 [term 1 FOLLOWER]: Refusing update from remote peer 533ccda3bae04fb3969458d59827aa9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:06:34.990446 29064 raft_consensus.cc:1261] T 274562ff4e3d4b54aada1a5b55d278b0 P 8cfb62e1db65439b9045f0af742caa83 [term 1 FOLLOWER]: Refusing update from remote peer 533ccda3bae04fb3969458d59827aa9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:06:34.991163 29402 consensus_queue.cc:1035] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ffb33985044eb9bf409474f946e289" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 40409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:06:34.991326 29405 consensus_queue.cc:1035] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cfb62e1db65439b9045f0af742caa83" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 39719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:06:34.999917 29409 mvcc.cc:205] Tried to move back new op lower bound from 7315603435473809408 to 7315603435294302208. Current Snapshot: MvccSnapshot[applied={T|T < 7315603435473809408}]
I20260806 17:06:35.002233 29410 mvcc.cc:205] Tried to move back new op lower bound from 7315603435473809408 to 7315603435294302208. Current Snapshot: MvccSnapshot[applied={T|T < 7315603435473809408}]
I20260806 17:06:35.002914 29408 mvcc.cc:205] Tried to move back new op lower bound from 7315603435473809408 to 7315603435294302208. Current Snapshot: MvccSnapshot[applied={T|T < 7315603435473809408 or (T in {7315603435473809408})}]
I20260806 17:06:35.009359 28921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:06:35.009564 28921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:06:35.012723 28921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a452c469da9f4687bd3fa48de8c7c6d0: Sending DeleteTablet for 3 replicas of tablet 274562ff4e3d4b54aada1a5b55d278b0
I20260806 17:06:35.013713 29310 tablet_service.cc:1514] Processing DeleteTablet for tablet 274562ff4e3d4b54aada1a5b55d278b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:06:35 UTC) from {username='slave'} at 127.0.0.1:35124
I20260806 17:06:35.013731 29177 tablet_service.cc:1514] Processing DeleteTablet for tablet 274562ff4e3d4b54aada1a5b55d278b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:06:35 UTC) from {username='slave'} at 127.0.0.1:52364
I20260806 17:06:35.014037 29044 tablet_service.cc:1514] Processing DeleteTablet for tablet 274562ff4e3d4b54aada1a5b55d278b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:06:35 UTC) from {username='slave'} at 127.0.0.1:35010
I20260806 17:06:35.014411 29417 tablet_replica.cc:330] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289: stopping tablet replica
I20260806 17:06:35.014498 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 28982
I20260806 17:06:35.014508 29419 tablet_replica.cc:330] T 274562ff4e3d4b54aada1a5b55d278b0 P 8cfb62e1db65439b9045f0af742caa83: stopping tablet replica
I20260806 17:06:35.014878 29417 raft_consensus.cc:2229] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:06:35.015272 29417 raft_consensus.cc:2258] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:06:35.016005 29418 tablet_replica.cc:330] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b: stopping tablet replica
I20260806 17:06:35.016551 29418 raft_consensus.cc:2229] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:06:35.017050 29418 raft_consensus.cc:2258] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:06:35.019762 29417 ts_tablet_manager.cc:1905] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:06:35.019761 29418 ts_tablet_manager.cc:1905] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:06:35.023067 29418 ts_tablet_manager.cc:1918] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:06:35.023276 29418 log.cc:1196] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/274562ff4e3d4b54aada1a5b55d278b0
I20260806 17:06:35.023893 29418 ts_tablet_manager.cc:1939] T 274562ff4e3d4b54aada1a5b55d278b0 P 533ccda3bae04fb3969458d59827aa9b: Deleting consensus metadata
I20260806 17:06:35.023880 29417 ts_tablet_manager.cc:1918] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:06:35.024056 29417 log.cc:1196] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/274562ff4e3d4b54aada1a5b55d278b0
I20260806 17:06:35.024467 29417 ts_tablet_manager.cc:1939] T 274562ff4e3d4b54aada1a5b55d278b0 P c2ffb33985044eb9bf409474f946e289: Deleting consensus metadata
W20260806 17:06:35.024717 28907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.111.65:39719 (error 108)
I20260806 17:06:35.025591 28906 catalog_manager.cc:4841] TS 533ccda3bae04fb3969458d59827aa9b (127.9.111.66:34547): tablet 274562ff4e3d4b54aada1a5b55d278b0 (table test-table [id=dbd9e8ba26b1402f87d9a4bfaa1e25f9]) successfully deleted
I20260806 17:06:35.025780 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 29115
I20260806 17:06:35.027756 28905 catalog_manager.cc:4841] TS c2ffb33985044eb9bf409474f946e289 (127.9.111.67:40409): tablet 274562ff4e3d4b54aada1a5b55d278b0 (table test-table [id=dbd9e8ba26b1402f87d9a4bfaa1e25f9]) successfully deleted
I20260806 17:06:35.028638 29389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:35.028284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:39719 (local address 127.0.0.1:35018)
0806 17:06:35.028479 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:35.028555 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.65:39719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260806 17:06:35.028882 28907 catalog_manager.cc:4584] TS 8cfb62e1db65439b9045f0af742caa83 (127.9.111.65:39719): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 274562ff4e3d4b54aada1a5b55d278b0: Network error: Client connection negotiation failed: client connection to 127.9.111.65:39719: connect: Connection refused (error 111)
I20260806 17:06:35.033852 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 29248
I20260806 17:06:35.041508 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 28889
2026-08-06T17:06:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1137 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:06:35 dist-test-slave-69rg krb5kdc[29423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:06:35 dist-test-slave-69rg krb5kdc[29423](info): set up 2 sockets
Aug 06 17:06:35 dist-test-slave-69rg krb5kdc[29423](info): commencing operation
krb5kdc: starting...
W20260806 17:06:36.124984 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:06:36 dist-test-slave-69rg krb5kdc[29423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:06:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:06:36.262954 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:45567
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:45567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:36.381716 29436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:36.382117 29436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:36.382213 29436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:36.386626 29436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:36.386736 29436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:36.386773 29436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:36.386811 29436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:36.386847 29436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:36.392371 29436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:45567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:45567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:36.393533 29436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:36.394588 29436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:36.403575 29441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:36.403782 29442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:36.403748 29444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:36.404315 29436 server_base.cc:1034] running on GCE node
I20260806 17:06:36.405053 29436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:36.406533 29436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:36.415663 29436 hybrid_clock.cc:648] HybridClock initialized: now 1786035996415619 us; error 50 us; skew 500 ppm
Aug 06 17:06:36 dist-test-slave-69rg krb5kdc[29423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035996, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:36.419596 29436 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:06:36.420583 29436 webserver.cc:460] Webserver started at http://127.9.111.126:37725/ using document root <none> and password file <none>
I20260806 17:06:36.421178 29436 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:36.421310 29436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:36.421557 29436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:36.423744 29436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "a3110a4e3cf24280a2833184fc6e4cc7"
format_stamp: "Formatted at 2026-08-06 17:06:36 on dist-test-slave-69rg"
server_key: "2296e10a71180c7dc56b408a30ef5cfb"
server_key_iv: "1981ff84a9a0963403915a5453a0145f"
server_key_version: "encryptionkey@0"
I20260806 17:06:36.424340 29436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "a3110a4e3cf24280a2833184fc6e4cc7"
format_stamp: "Formatted at 2026-08-06 17:06:36 on dist-test-slave-69rg"
server_key: "2296e10a71180c7dc56b408a30ef5cfb"
server_key_iv: "1981ff84a9a0963403915a5453a0145f"
server_key_version: "encryptionkey@0"
I20260806 17:06:36.428397 29436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 17:06:36.431146 29451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:36.432525 29436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:06:36.432665 29436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "a3110a4e3cf24280a2833184fc6e4cc7"
format_stamp: "Formatted at 2026-08-06 17:06:36 on dist-test-slave-69rg"
server_key: "2296e10a71180c7dc56b408a30ef5cfb"
server_key_iv: "1981ff84a9a0963403915a5453a0145f"
server_key_version: "encryptionkey@0"
I20260806 17:06:36.432798 29436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:36.446889 29436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:36.450683 29436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:36.450903 29436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:36.459072 29436 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:45567
I20260806 17:06:36.459085 29504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:45567 every 8 connection(s)
I20260806 17:06:36.460047 29436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:36.460821 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 29436
I20260806 17:06:36.460964 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:36.461246 9661 external_mini_cluster.cc:1428] Setting key 08bccb205b322657ef416aa01ac576d1
I20260806 17:06:36.464663 29505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 17:06:36 dist-test-slave-69rg krb5kdc[29423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:06:36.473292 29505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7: Bootstrap starting.
I20260806 17:06:36.475045 29508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:36.462644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41830 (local address 127.9.111.126:45567)
0806 17:06:36.463064 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:36.463092 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:36.463125 (+ 33us) server_negotiation.cc:407] Connection header received
0806 17:06:36.463829 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:36.463846 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:36.464154 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:36.464489 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:36.465329 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:36.466412 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:36.467253 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:36.467718 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:36.467906 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:36.470430 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:36.470435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:36.470446 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:36.470473 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:36.472154 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:36.472615 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:36.472620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:36.472624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:36.472732 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:36.473021 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:36.473025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:36.473027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:36.473451 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:36.473659 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:36.473952 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 17:06:36.474360 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":161,"threads_started":1}
I20260806 17:06:36.476545 29505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:36.477578 29505 log.cc:824] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:36.480182 29505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7: No bootstrap required, opened a new log
I20260806 17:06:36.486428 29505 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3110a4e3cf24280a2833184fc6e4cc7" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45567 } }
I20260806 17:06:36.486802 29505 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:36.486889 29505 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3110a4e3cf24280a2833184fc6e4cc7, State: Initialized, Role: FOLLOWER
I20260806 17:06:36.487468 29505 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3110a4e3cf24280a2833184fc6e4cc7" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45567 } }
I20260806 17:06:36.487665 29505 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:36.487761 29505 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:36.487882 29505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:36.489464 29505 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3110a4e3cf24280a2833184fc6e4cc7" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45567 } }
I20260806 17:06:36.489957 29505 leader_election.cc:302] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3110a4e3cf24280a2833184fc6e4cc7; no voters:
I20260806 17:06:36.490437 29505 leader_election.cc:288] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:36.490559 29510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:36.490844 29510 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7 [term 1 LEADER]: Becoming Leader. State: Replica: a3110a4e3cf24280a2833184fc6e4cc7, State: Running, Role: LEADER
I20260806 17:06:36.491219 29510 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3110a4e3cf24280a2833184fc6e4cc7" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45567 } }
I20260806 17:06:36.492125 29505 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:36.492370 29510 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3110a4e3cf24280a2833184fc6e4cc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3110a4e3cf24280a2833184fc6e4cc7" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45567 } } }
I20260806 17:06:36.492484 29510 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:36.492915 29510 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3110a4e3cf24280a2833184fc6e4cc7. Latest consensus state: current_term: 1 leader_uuid: "a3110a4e3cf24280a2833184fc6e4cc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3110a4e3cf24280a2833184fc6e4cc7" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45567 } } }
I20260806 17:06:36.493018 29510 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:36.493356 29514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:36.496577 29514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:36.508108 29514 catalog_manager.cc:1347] Generated new cluster ID: 65d9a4d5a361428e9ea404ad4d933126
I20260806 17:06:36.508195 29514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:36.523265 29514 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:36.524216 29514 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:36.528554 29514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7: Generated new TSK 0
I20260806 17:06:36.529184 29514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.111.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
Entry for principal kudu/127.9.111.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab.
I20260806 17:06:36.566049 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:45567
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:36.684923 29531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:36.685268 29531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:36.685355 29531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:36.689533 29531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:36.689648 29531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:36.689754 29531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:06:36.695492 29531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.65.keytab
--principal=kudu/127.9.111.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:45567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:36.696631 29531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:36.697670 29531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:36.707712 29537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:36.707746 29536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:36.707691 29539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:36.707959 29531 server_base.cc:1034] running on GCE node
I20260806 17:06:36.708577 29531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:36.709306 29531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:36.710536 29531 hybrid_clock.cc:648] HybridClock initialized: now 1786035996710515 us; error 32 us; skew 500 ppm
Aug 06 17:06:36 dist-test-slave-69rg krb5kdc[29423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035996, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:36.713758 29531 init.cc:377] Logged in from keytab as kudu/127.9.111.65@KRBTEST.COM (short username kudu)
I20260806 17:06:36.714851 29531 webserver.cc:460] Webserver started at http://127.9.111.65:39485/ using document root <none> and password file <none>
I20260806 17:06:36.715464 29531 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:36.715564 29531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:36.715821 29531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:36.718122 29531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "d9b662ecb86945899c736e13e83874d1"
format_stamp: "Formatted at 2026-08-06 17:06:36 on dist-test-slave-69rg"
server_key: "d51e1c9160da453476a4ce750bdd68f3"
server_key_iv: "997469ad9ea90fa97e43754a783e6093"
server_key_version: "encryptionkey@0"
I20260806 17:06:36.718731 29531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "d9b662ecb86945899c736e13e83874d1"
format_stamp: "Formatted at 2026-08-06 17:06:36 on dist-test-slave-69rg"
server_key: "d51e1c9160da453476a4ce750bdd68f3"
server_key_iv: "997469ad9ea90fa97e43754a783e6093"
server_key_version: "encryptionkey@0"
I20260806 17:06:36.722681 29531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 17:06:36.725347 29546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:36.726740 29531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 17:06:36.726913 29531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "d9b662ecb86945899c736e13e83874d1"
format_stamp: "Formatted at 2026-08-06 17:06:36 on dist-test-slave-69rg"
server_key: "d51e1c9160da453476a4ce750bdd68f3"
server_key_iv: "997469ad9ea90fa97e43754a783e6093"
server_key_version: "encryptionkey@0"
I20260806 17:06:36.727051 29531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:36.745182 29531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:36.748627 29531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:36.748878 29531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:36.749544 29531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:36.750530 29531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:36.750609 29531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:36.750685 29531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:36.750722 29531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:36.760525 29531 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:46731
I20260806 17:06:36.760545 29664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:46731 every 8 connection(s)
I20260806 17:06:36.761500 29531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:06:36.764508 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 29531
I20260806 17:06:36.764678 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:06:36.765003 9661 external_mini_cluster.cc:1428] Setting key ff3436bb4af06f1e5c8ee45f21f742d9
Aug 06 17:06:36 dist-test-slave-69rg krb5kdc[29423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035996, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.65@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.111.66@KRBTEST.COM; defaulting to no policy
I20260806 17:06:36.778612 29508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:36.763646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:45063 (local address 127.9.111.126:45567)
0806 17:06:36.763808 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:36.763812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:36.764862 (+ 1050us) server_negotiation.cc:407] Connection header received
0806 17:06:36.766713 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:36.766717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:36.766776 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:36.766881 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:36.768138 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:36.769084 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:36.770959 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:36.771200 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:36.771297 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:36.774490 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:36.774494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:36.774496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:36.774523 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:36.776104 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:36.776679 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:36.776684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:36.776691 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:36.776772 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:36.777147 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:36.777157 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:36.777159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:36.777418 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:36.777544 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:36.778218 (+ 674us) server_negotiation.cc:299] Negotiation successful
0806 17:06:36.778458 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.66@KRBTEST.COM" created.
I20260806 17:06:36.779654 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:36.764034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:45567 (local address 127.9.111.65:45063)
0806 17:06:36.764665 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:36.764731 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:36.766388 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:36.767062 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:36.767069 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:36.767463 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:36.767972 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:36.767986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:36.769236 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:36.769242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:36.770816 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:36.770831 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:36.771370 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:36.771377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:36.771617 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:36.772426 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:06:36.772453 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:36.774288 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:36.776253 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:36.776259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:36.776277 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:36.776559 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:36.776878 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:36.776881 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:36.776884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:36.777039 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:36.777567 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:36.777573 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:36.777919 (+ 346us) client_negotiation.cc:769] Sending connection context
0806 17:06:36.778116 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 17:06:36.778448 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":186,"threads_started":1}
I20260806 17:06:36.781180 29665 heartbeater.cc:347] Connected to a master server at 127.9.111.126:45567
I20260806 17:06:36.781447 29665 heartbeater.cc:464] Registering TS with master...
I20260806 17:06:36.782104 29665 heartbeater.cc:511] Master 127.9.111.126:45567 requested a full tablet report, sending...
I20260806 17:06:36.783851 29469 ts_manager.cc:194] Registered new tserver with Master: d9b662ecb86945899c736e13e83874d1 (127.9.111.65:46731)
I20260806 17:06:36.785673 29469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.65@KRBTEST.COM'} at 127.9.111.65:45063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
Entry for principal kudu/127.9.111.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab.
I20260806 17:06:36.799213 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:45567
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:36.919798 29670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:36.920091 29670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:36.920138 29670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:36.924293 29670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:36.924371 29670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:36.924432 29670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:06:36.930055 29670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.66.keytab
--principal=kudu/127.9.111.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:45567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:36.931303 29670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:36.932415 29670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:36.942588 29675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:36.942579 29678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:36.942754 29670 server_base.cc:1034] running on GCE node
W20260806 17:06:36.942948 29676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:36.943406 29670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:36.944110 29670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:36.945326 29670 hybrid_clock.cc:648] HybridClock initialized: now 1786035996945297 us; error 41 us; skew 500 ppm
Aug 06 17:06:36 dist-test-slave-69rg krb5kdc[29423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035996, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:36.948391 29670 init.cc:377] Logged in from keytab as kudu/127.9.111.66@KRBTEST.COM (short username kudu)
I20260806 17:06:36.949299 29670 webserver.cc:460] Webserver started at http://127.9.111.66:37323/ using document root <none> and password file <none>
I20260806 17:06:36.949926 29670 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:36.950026 29670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:36.950258 29670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:36.952358 29670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "383c7678afb9409b923d077c40afe1bb"
format_stamp: "Formatted at 2026-08-06 17:06:36 on dist-test-slave-69rg"
server_key: "54edc22c8a734b688e144bdb7046e3da"
server_key_iv: "34be0701ba47db0362a6c8913bf214f3"
server_key_version: "encryptionkey@0"
I20260806 17:06:36.952945 29670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "383c7678afb9409b923d077c40afe1bb"
format_stamp: "Formatted at 2026-08-06 17:06:36 on dist-test-slave-69rg"
server_key: "54edc22c8a734b688e144bdb7046e3da"
server_key_iv: "34be0701ba47db0362a6c8913bf214f3"
server_key_version: "encryptionkey@0"
I20260806 17:06:36.956898 29670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 17:06:36.959474 29685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:36.960789 29670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 17:06:36.960929 29670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "383c7678afb9409b923d077c40afe1bb"
format_stamp: "Formatted at 2026-08-06 17:06:36 on dist-test-slave-69rg"
server_key: "54edc22c8a734b688e144bdb7046e3da"
server_key_iv: "34be0701ba47db0362a6c8913bf214f3"
server_key_version: "encryptionkey@0"
I20260806 17:06:36.961046 29670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:36.978055 29670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:36.981540 29670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:36.981783 29670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:36.982498 29670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:36.983378 29670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:36.983448 29670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:36.983518 29670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:36.983565 29670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:36.992884 29670 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:36769
I20260806 17:06:36.992902 29800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:36769 every 8 connection(s)
I20260806 17:06:36.994040 29670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:06:36.997709 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 29670
I20260806 17:06:36.997885 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:06:36.998203 9661 external_mini_cluster.cc:1428] Setting key 7ec7e806a0596142a43e61f15a6cc9f0
Aug 06 17:06:37 dist-test-slave-69rg krb5kdc[29423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035996, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.66@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.111.67@KRBTEST.COM; defaulting to no policy
I20260806 17:06:37.010568 29508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:36.995871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:36495 (local address 127.9.111.126:45567)
0806 17:06:36.996021 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:36.996024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:36.996841 (+ 817us) server_negotiation.cc:407] Connection header received
0806 17:06:36.998652 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:36.998657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:36.998720 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:36.998818 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:37.000361 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.000870 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.002493 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.002887 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.003018 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:37.006182 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:37.006188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:37.006195 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:37.006236 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:37.008059 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:37.008666 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:37.008670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:37.008674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:37.008777 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:37.009141 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:37.009147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:37.009151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:37.009453 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:37.009577 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:37.010207 (+ 630us) server_negotiation.cc:299] Negotiation successful
0806 17:06:37.010378 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 17:06:37.011420 29803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:36.996242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:45567 (local address 127.9.111.66:36495)
0806 17:06:36.996675 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:36.996717 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:36.998393 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:36.999017 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:36.999031 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:36.999457 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:37.000187 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:37.000212 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.001001 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.001007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:37.002361 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:37.002372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.003050 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.003053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:37.003263 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:37.003983 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:06:37.004010 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:37.005960 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:37.008211 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:37.008217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:37.008232 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:37.008530 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:37.008876 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:37.008879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:37.008880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:37.009026 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:37.009568 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:37.009573 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:37.009913 (+ 340us) client_negotiation.cc:769] Sending connection context
0806 17:06:37.010080 (+ 167us) client_negotiation.cc:240] Negotiation successful
0806 17:06:37.010432 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":112,"threads_started":1}
I20260806 17:06:37.013044 29801 heartbeater.cc:347] Connected to a master server at 127.9.111.126:45567
I20260806 17:06:37.013367 29801 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.67@KRBTEST.COM" created.
I20260806 17:06:37.014104 29801 heartbeater.cc:511] Master 127.9.111.126:45567 requested a full tablet report, sending...
I20260806 17:06:37.015511 29469 ts_manager.cc:194] Registered new tserver with Master: 383c7678afb9409b923d077c40afe1bb (127.9.111.66:36769)
I20260806 17:06:37.016415 29469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.66@KRBTEST.COM'} at 127.9.111.66:36495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
Entry for principal kudu/127.9.111.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab.
I20260806 17:06:37.033679 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:45567
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:37.148926 29807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:37.149225 29807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:37.149281 29807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:37.153609 29807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:37.153692 29807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:37.153753 29807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:06:37.159516 29807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.67.keytab
--principal=kudu/127.9.111.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:45567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:37.160624 29807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:37.161607 29807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:37.171562 29812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:37.171550 29813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:37.171556 29815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:37.171783 29807 server_base.cc:1034] running on GCE node
I20260806 17:06:37.172344 29807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:37.173038 29807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:37.174223 29807 hybrid_clock.cc:648] HybridClock initialized: now 1786035997174209 us; error 31 us; skew 500 ppm
Aug 06 17:06:37 dist-test-slave-69rg krb5kdc[29423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035997, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:37.177093 29807 init.cc:377] Logged in from keytab as kudu/127.9.111.67@KRBTEST.COM (short username kudu)
I20260806 17:06:37.178121 29807 webserver.cc:460] Webserver started at http://127.9.111.67:41945/ using document root <none> and password file <none>
I20260806 17:06:37.178728 29807 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:37.178826 29807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:37.179065 29807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:37.181200 29807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "67f241431cf14e28981e6cafdd2e1e0c"
format_stamp: "Formatted at 2026-08-06 17:06:37 on dist-test-slave-69rg"
server_key: "0afdc13eaca221e1dfcd0a3db2013787"
server_key_iv: "d5d848fbbab790cd1e5d3e0b952b9084"
server_key_version: "encryptionkey@0"
I20260806 17:06:37.181803 29807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "67f241431cf14e28981e6cafdd2e1e0c"
format_stamp: "Formatted at 2026-08-06 17:06:37 on dist-test-slave-69rg"
server_key: "0afdc13eaca221e1dfcd0a3db2013787"
server_key_iv: "d5d848fbbab790cd1e5d3e0b952b9084"
server_key_version: "encryptionkey@0"
I20260806 17:06:37.185702 29807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 17:06:37.188429 29822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:37.189750 29807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 17:06:37.189915 29807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "67f241431cf14e28981e6cafdd2e1e0c"
format_stamp: "Formatted at 2026-08-06 17:06:37 on dist-test-slave-69rg"
server_key: "0afdc13eaca221e1dfcd0a3db2013787"
server_key_iv: "d5d848fbbab790cd1e5d3e0b952b9084"
server_key_version: "encryptionkey@0"
I20260806 17:06:37.190042 29807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:37.218151 29807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:37.221889 29807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:37.222151 29807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:37.222838 29807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:37.223724 29807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:37.223804 29807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:37.223877 29807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:37.223922 29807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:37.234555 29807 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:35289
I20260806 17:06:37.234573 29939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:35289 every 8 connection(s)
I20260806 17:06:37.235571 29807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:06:37.241972 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 29807
I20260806 17:06:37.242130 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:06:37.242470 9661 external_mini_cluster.cc:1428] Setting key 20d7eb1486880bcbf5e72017982b1dad
Aug 06 17:06:37 dist-test-slave-69rg krb5kdc[29423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035997, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.67@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:06:37.250357 29508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:37.237672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:38399 (local address 127.9.111.126:45567)
0806 17:06:37.237845 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:37.237849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:37.238843 (+ 994us) server_negotiation.cc:407] Connection header received
0806 17:06:37.240152 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:37.240155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:37.240226 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:37.240365 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:37.241520 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.242186 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.243612 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.243939 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.244052 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:37.246815 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:37.246818 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:37.246820 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:37.246845 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:37.248271 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:37.248821 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:37.248824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:37.248825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:37.248928 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:37.249241 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:37.249244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:37.249245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:37.249449 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:37.249535 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:37.250097 (+ 562us) server_negotiation.cc:299] Negotiation successful
0806 17:06:37.250230 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 17:06:37.251152 29942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:37.238132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:45567 (local address 127.9.111.67:38399)
0806 17:06:37.238670 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:37.238713 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:37.239946 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:37.240537 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:37.240543 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:37.240911 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:37.241379 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:37.241391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.242322 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.242326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:37.243446 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:37.243473 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.244108 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.244113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:37.244302 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:37.245075 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:06:37.245092 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:37.246638 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:37.248419 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:37.248425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:37.248438 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:37.248714 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:37.249012 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:06:37.249014 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:06:37.249016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:06:37.249148 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:06:37.249537 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:37.249543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:37.249811 (+ 268us) client_negotiation.cc:769] Sending connection context
0806 17:06:37.250029 (+ 218us) client_negotiation.cc:240] Negotiation successful
0806 17:06:37.250316 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":183,"threads_started":1}
I20260806 17:06:37.252334 29940 heartbeater.cc:347] Connected to a master server at 127.9.111.126:45567
I20260806 17:06:37.252558 29940 heartbeater.cc:464] Registering TS with master...
I20260806 17:06:37.253027 29940 heartbeater.cc:511] Master 127.9.111.126:45567 requested a full tablet report, sending...
I20260806 17:06:37.254076 29469 ts_manager.cc:194] Registered new tserver with Master: 67f241431cf14e28981e6cafdd2e1e0c (127.9.111.67:35289)
I20260806 17:06:37.254643 29469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.111.67@KRBTEST.COM'} at 127.9.111.67:38399
I20260806 17:06:37.256655 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 17:06:37.266304 29508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:37.258682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41844 (local address 127.9.111.126:45567)
0806 17:06:37.258843 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:37.258848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:37.259034 (+ 186us) server_negotiation.cc:407] Connection header received
0806 17:06:37.259134 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:37.259138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:37.259189 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:37.259253 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:37.259911 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.260428 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.261272 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.261487 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.261577 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:37.263018 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:37.263021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:37.263023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:37.263045 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:37.264511 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:37.265002 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:37.265004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:37.265006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:37.265081 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:37.265385 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:37.265388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:37.265389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:37.265603 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:37.265701 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:37.265970 (+ 269us) server_negotiation.cc:299] Negotiation successful
0806 17:06:37.266073 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 17:06:37.269235 29469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:06:37.272353 29469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:06:37.286952 29953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:37.280917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:36769 (local address 127.0.0.1:56922)
0806 17:06:37.282120 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:37.282135 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:37.282228 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:37.282853 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:37.282856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:37.282873 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:37.282971 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:37.282977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.284396 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.284400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:37.285502 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:37.285509 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.286608 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.286614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:37.286698 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:37.286718 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 17:06:37.286803 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 17:06:37.286860 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1138,"thread_start_us":109,"threads_started":1}
I20260806 17:06:37.287225 29952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:37.281204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:46731 (local address 127.0.0.1:33486)
0806 17:06:37.282477 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:37.282490 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:37.282587 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:37.283096 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:37.283099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:37.283116 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:37.283184 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:37.283190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.284484 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.284491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:37.285617 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:37.285628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.286814 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.286817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:37.286896 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:37.286914 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 17:06:37.287002 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 17:06:37.287077 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1211,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260806 17:06:37.287289 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:37.281633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33486 (local address 127.9.111.65:46731)
0806 17:06:37.281997 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:37.282007 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:37.282692 (+ 685us) server_negotiation.cc:407] Connection header received
0806 17:06:37.282773 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:37.282781 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:37.282948 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:37.283079 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:37.283295 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.284371 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.285753 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.286547 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.286639 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:37.286761 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:37.287043 (+ 282us) server_negotiation.cc:299] Negotiation successful
0806 17:06:37.287151 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":146,"threads_started":1}
I20260806 17:06:37.286952 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:37.280847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:35289 (local address 127.0.0.1:56054)
0806 17:06:37.281427 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:37.281482 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:37.281609 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:37.282629 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:37.282635 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:37.282658 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:37.282853 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:37.282861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.284003 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.284007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:37.285333 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:37.285344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.286365 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.286369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:37.286483 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:37.286521 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 17:06:37.286633 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 17:06:37.286784 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":100,"threads_started":1}
I20260806 17:06:37.287607 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:37.282125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.9.111.66:36769)
0806 17:06:37.282396 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:37.282408 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:37.282430 (+ 22us) server_negotiation.cc:407] Connection header received
0806 17:06:37.282505 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:37.282514 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:37.282702 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:37.282863 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:37.283062 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.284251 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.285614 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.286463 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.286586 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:37.286732 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:37.287332 (+ 600us) server_negotiation.cc:299] Negotiation successful
0806 17:06:37.287453 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":107,"threads_started":1}
I20260806 17:06:37.287997 29954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:37.281570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56054 (local address 127.9.111.67:35289)
0806 17:06:37.282141 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:37.282157 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:37.282183 (+ 26us) server_negotiation.cc:407] Connection header received
0806 17:06:37.282278 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:37.282284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:37.282470 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:37.282637 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:37.282990 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.283862 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.285466 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.286242 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.286363 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:37.286525 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:37.287751 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0806 17:06:37.287862 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":147,"threads_started":1}
I20260806 17:06:37.290453 29734 tablet_service.cc:1467] Processing CreateTablet for tablet 96bced8ce9fc4fb2b9661abf099cf3b9 (DEFAULT_TABLE table=test-table [id=7ec467dde5fa44e2a6079b7ce3c5ff6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:06:37.290457 29594 tablet_service.cc:1467] Processing CreateTablet for tablet 96bced8ce9fc4fb2b9661abf099cf3b9 (DEFAULT_TABLE table=test-table [id=7ec467dde5fa44e2a6079b7ce3c5ff6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:06:37.290455 29872 tablet_service.cc:1467] Processing CreateTablet for tablet 96bced8ce9fc4fb2b9661abf099cf3b9 (DEFAULT_TABLE table=test-table [id=7ec467dde5fa44e2a6079b7ce3c5ff6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:06:37.291460 29734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96bced8ce9fc4fb2b9661abf099cf3b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:37.291486 29872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96bced8ce9fc4fb2b9661abf099cf3b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:37.291481 29594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96bced8ce9fc4fb2b9661abf099cf3b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:37.298736 29957 tablet_bootstrap.cc:492] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb: Bootstrap starting.
I20260806 17:06:37.299798 29959 tablet_bootstrap.cc:492] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c: Bootstrap starting.
I20260806 17:06:37.299854 29958 tablet_bootstrap.cc:492] T 96bced8ce9fc4fb2b9661abf099cf3b9 P d9b662ecb86945899c736e13e83874d1: Bootstrap starting.
I20260806 17:06:37.301450 29959 tablet_bootstrap.cc:654] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:37.301620 29957 tablet_bootstrap.cc:654] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:37.301908 29958 tablet_bootstrap.cc:654] T 96bced8ce9fc4fb2b9661abf099cf3b9 P d9b662ecb86945899c736e13e83874d1: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:37.302597 29959 log.cc:824] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:37.302593 29957 log.cc:824] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:37.302597 29958 log.cc:824] T 96bced8ce9fc4fb2b9661abf099cf3b9 P d9b662ecb86945899c736e13e83874d1: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:37.304873 29957 tablet_bootstrap.cc:492] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb: No bootstrap required, opened a new log
I20260806 17:06:37.304873 29958 tablet_bootstrap.cc:492] T 96bced8ce9fc4fb2b9661abf099cf3b9 P d9b662ecb86945899c736e13e83874d1: No bootstrap required, opened a new log
I20260806 17:06:37.305074 29958 ts_tablet_manager.cc:1397] T 96bced8ce9fc4fb2b9661abf099cf3b9 P d9b662ecb86945899c736e13e83874d1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260806 17:06:37.305074 29957 ts_tablet_manager.cc:1397] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260806 17:06:37.305097 29959 tablet_bootstrap.cc:492] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c: No bootstrap required, opened a new log
I20260806 17:06:37.305313 29959 ts_tablet_manager.cc:1397] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 17:06:37.308221 29959 raft_consensus.cc:348] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f241431cf14e28981e6cafdd2e1e0c" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35289 } } peers { permanent_uuid: "383c7678afb9409b923d077c40afe1bb" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36769 } } peers { permanent_uuid: "d9b662ecb86945899c736e13e83874d1" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 46731 } }
I20260806 17:06:37.308497 29959 raft_consensus.cc:374] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:37.308550 29959 raft_consensus.cc:729] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67f241431cf14e28981e6cafdd2e1e0c, State: Initialized, Role: FOLLOWER
I20260806 17:06:37.308997 29959 consensus_queue.cc:260] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f241431cf14e28981e6cafdd2e1e0c" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35289 } } peers { permanent_uuid: "383c7678afb9409b923d077c40afe1bb" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36769 } } peers { permanent_uuid: "d9b662ecb86945899c736e13e83874d1" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 46731 } }
I20260806 17:06:37.309108 29957 raft_consensus.cc:348] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f241431cf14e28981e6cafdd2e1e0c" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35289 } } peers { permanent_uuid: "383c7678afb9409b923d077c40afe1bb" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36769 } } peers { permanent_uuid: "d9b662ecb86945899c736e13e83874d1" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 46731 } }
I20260806 17:06:37.309391 29957 raft_consensus.cc:374] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:37.309451 29957 raft_consensus.cc:729] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383c7678afb9409b923d077c40afe1bb, State: Initialized, Role: FOLLOWER
I20260806 17:06:37.309850 29940 heartbeater.cc:503] Master 127.9.111.126:45567 was elected leader, sending a full tablet report...
I20260806 17:06:37.309934 29957 consensus_queue.cc:260] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f241431cf14e28981e6cafdd2e1e0c" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35289 } } peers { permanent_uuid: "383c7678afb9409b923d077c40afe1bb" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36769 } } peers { permanent_uuid: "d9b662ecb86945899c736e13e83874d1" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 46731 } }
I20260806 17:06:37.310141 29959 ts_tablet_manager.cc:1428] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 17:06:37.309557 29958 raft_consensus.cc:348] T 96bced8ce9fc4fb2b9661abf099cf3b9 P d9b662ecb86945899c736e13e83874d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f241431cf14e28981e6cafdd2e1e0c" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35289 } } peers { permanent_uuid: "383c7678afb9409b923d077c40afe1bb" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36769 } } peers { permanent_uuid: "d9b662ecb86945899c736e13e83874d1" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 46731 } }
I20260806 17:06:37.310418 29958 raft_consensus.cc:374] T 96bced8ce9fc4fb2b9661abf099cf3b9 P d9b662ecb86945899c736e13e83874d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:37.310473 29958 raft_consensus.cc:729] T 96bced8ce9fc4fb2b9661abf099cf3b9 P d9b662ecb86945899c736e13e83874d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b662ecb86945899c736e13e83874d1, State: Initialized, Role: FOLLOWER
I20260806 17:06:37.310714 29801 heartbeater.cc:503] Master 127.9.111.126:45567 was elected leader, sending a full tablet report...
I20260806 17:06:37.310869 29957 ts_tablet_manager.cc:1428] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 17:06:37.310917 29958 consensus_queue.cc:260] T 96bced8ce9fc4fb2b9661abf099cf3b9 P d9b662ecb86945899c736e13e83874d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f241431cf14e28981e6cafdd2e1e0c" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35289 } } peers { permanent_uuid: "383c7678afb9409b923d077c40afe1bb" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36769 } } peers { permanent_uuid: "d9b662ecb86945899c736e13e83874d1" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 46731 } }
I20260806 17:06:37.311568 29665 heartbeater.cc:503] Master 127.9.111.126:45567 was elected leader, sending a full tablet report...
I20260806 17:06:37.312438 29958 ts_tablet_manager.cc:1428] T 96bced8ce9fc4fb2b9661abf099cf3b9 P d9b662ecb86945899c736e13e83874d1: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260806 17:06:37.476615 29964 raft_consensus.cc:482] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:06:37.476805 29964 raft_consensus.cc:504] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f241431cf14e28981e6cafdd2e1e0c" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35289 } } peers { permanent_uuid: "383c7678afb9409b923d077c40afe1bb" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36769 } } peers { permanent_uuid: "d9b662ecb86945899c736e13e83874d1" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 46731 } }
I20260806 17:06:37.477979 29964 leader_election.cc:288] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67f241431cf14e28981e6cafdd2e1e0c (127.9.111.67:35289), d9b662ecb86945899c736e13e83874d1 (127.9.111.65:46731)
I20260806 17:06:37.482105 29803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:37.478173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:35289 (local address 127.9.111.66:58819)
0806 17:06:37.478356 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:37.478372 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:37.478474 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:37.478801 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:37.478803 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:37.478817 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:37.478950 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:37.478956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.479720 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.479723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:37.480747 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:37.480756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.481623 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.481630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:37.481740 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:37.481765 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 17:06:37.481882 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 17:06:37.481956 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":19}
I20260806 17:06:37.482115 29954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:37.478349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:58819 (local address 127.9.111.67:35289)
0806 17:06:37.478524 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:37.478530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:37.478547 (+ 17us) server_negotiation.cc:407] Connection header received
0806 17:06:37.478632 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:37.478641 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:37.478688 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:37.478783 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:37.479086 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.479610 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.480888 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.481510 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.481599 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:37.481640 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:37.481934 (+ 294us) server_negotiation.cc:299] Negotiation successful
0806 17:06:37.482008 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 17:06:37.482719 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:37.478543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:56109 (local address 127.9.111.65:46731)
0806 17:06:37.478669 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:37.478675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:37.479041 (+ 366us) server_negotiation.cc:407] Connection header received
0806 17:06:37.479095 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:37.479097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:37.479136 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:37.479219 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:37.479494 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.480193 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.481445 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.482085 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.482209 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:37.482261 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:37.482522 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 17:06:37.482604 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 17:06:37.482765 29966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:37.478351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:46731 (local address 127.9.111.66:56109)
0806 17:06:37.478669 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:37.478682 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:37.478802 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:37.479238 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:37.479241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:37.479256 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:37.479376 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:37.479381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.480311 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.480314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:37.481317 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:37.481327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.482246 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.482251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:37.482377 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:37.482401 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 17:06:37.482493 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 17:06:37.482570 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":2,"thread_start_us":111,"threads_started":1}
I20260806 17:06:37.482992 29893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96bced8ce9fc4fb2b9661abf099cf3b9" candidate_uuid: "383c7678afb9409b923d077c40afe1bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67f241431cf14e28981e6cafdd2e1e0c" is_pre_election: true
I20260806 17:06:37.483362 29893 raft_consensus.cc:2454] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 383c7678afb9409b923d077c40afe1bb in term 0.
I20260806 17:06:37.483403 29615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96bced8ce9fc4fb2b9661abf099cf3b9" candidate_uuid: "383c7678afb9409b923d077c40afe1bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b662ecb86945899c736e13e83874d1" is_pre_election: true
I20260806 17:06:37.483700 29615 raft_consensus.cc:2454] T 96bced8ce9fc4fb2b9661abf099cf3b9 P d9b662ecb86945899c736e13e83874d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 383c7678afb9409b923d077c40afe1bb in term 0.
I20260806 17:06:37.483959 29687 leader_election.cc:302] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 383c7678afb9409b923d077c40afe1bb, 67f241431cf14e28981e6cafdd2e1e0c; no voters:
I20260806 17:06:37.484303 29964 raft_consensus.cc:2781] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:06:37.484402 29964 raft_consensus.cc:482] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:06:37.484463 29964 raft_consensus.cc:3037] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:37.485504 29964 raft_consensus.cc:504] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f241431cf14e28981e6cafdd2e1e0c" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35289 } } peers { permanent_uuid: "383c7678afb9409b923d077c40afe1bb" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36769 } } peers { permanent_uuid: "d9b662ecb86945899c736e13e83874d1" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 46731 } }
I20260806 17:06:37.485879 29964 leader_election.cc:288] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [CANDIDATE]: Term 1 election: Requested vote from peers 67f241431cf14e28981e6cafdd2e1e0c (127.9.111.67:35289), d9b662ecb86945899c736e13e83874d1 (127.9.111.65:46731)
I20260806 17:06:37.486271 29893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96bced8ce9fc4fb2b9661abf099cf3b9" candidate_uuid: "383c7678afb9409b923d077c40afe1bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67f241431cf14e28981e6cafdd2e1e0c"
I20260806 17:06:37.486351 29615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96bced8ce9fc4fb2b9661abf099cf3b9" candidate_uuid: "383c7678afb9409b923d077c40afe1bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b662ecb86945899c736e13e83874d1"
I20260806 17:06:37.486431 29893 raft_consensus.cc:3037] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:37.486506 29615 raft_consensus.cc:3037] T 96bced8ce9fc4fb2b9661abf099cf3b9 P d9b662ecb86945899c736e13e83874d1 [term 0 FOLLOWER]: Advancing to term 1
W20260806 17:06:37.486573 29941 tablet.cc:2369] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:06:37.487556 29615 raft_consensus.cc:2454] T 96bced8ce9fc4fb2b9661abf099cf3b9 P d9b662ecb86945899c736e13e83874d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 383c7678afb9409b923d077c40afe1bb in term 1.
I20260806 17:06:37.487612 29893 raft_consensus.cc:2454] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 383c7678afb9409b923d077c40afe1bb in term 1.
I20260806 17:06:37.487915 29688 leader_election.cc:302] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 383c7678afb9409b923d077c40afe1bb, d9b662ecb86945899c736e13e83874d1; no voters:
I20260806 17:06:37.488103 29964 raft_consensus.cc:2781] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:37.488329 29964 raft_consensus.cc:686] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [term 1 LEADER]: Becoming Leader. State: Replica: 383c7678afb9409b923d077c40afe1bb, State: Running, Role: LEADER
I20260806 17:06:37.488670 29964 consensus_queue.cc:237] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f241431cf14e28981e6cafdd2e1e0c" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35289 } } peers { permanent_uuid: "383c7678afb9409b923d077c40afe1bb" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36769 } } peers { permanent_uuid: "d9b662ecb86945899c736e13e83874d1" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 46731 } }
I20260806 17:06:37.491976 29469 catalog_manager.cc:5495] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 383c7678afb9409b923d077c40afe1bb (127.9.111.66). New cstate: current_term: 1 leader_uuid: "383c7678afb9409b923d077c40afe1bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f241431cf14e28981e6cafdd2e1e0c" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "383c7678afb9409b923d077c40afe1bb" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 36769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9b662ecb86945899c736e13e83874d1" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 46731 } health_report { overall_health: UNKNOWN } } }
W20260806 17:06:37.495245 29802 tablet.cc:2369] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 17:06:37.513005 29666 tablet.cc:2369] T 96bced8ce9fc4fb2b9661abf099cf3b9 P d9b662ecb86945899c736e13e83874d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:06:37.527904 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:37.523963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.9.111.66:36769)
0806 17:06:37.524170 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:37.524175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:37.524189 (+ 14us) server_negotiation.cc:407] Connection header received
0806 17:06:37.524271 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:37.524274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:37.524338 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:37.524446 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:06:37.524784 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.525304 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.526357 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.526576 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.526667 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:37.527036 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:06:37.527516 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:06:37.527608 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:37.527729 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 17:06:37.527784 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 17:06:37.533974 29893 raft_consensus.cc:1261] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c [term 1 FOLLOWER]: Refusing update from remote peer 383c7678afb9409b923d077c40afe1bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:06:37.533974 29615 raft_consensus.cc:1261] T 96bced8ce9fc4fb2b9661abf099cf3b9 P d9b662ecb86945899c736e13e83874d1 [term 1 FOLLOWER]: Refusing update from remote peer 383c7678afb9409b923d077c40afe1bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:06:37.534727 29964 consensus_queue.cc:1035] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9b662ecb86945899c736e13e83874d1" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 46731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:06:37.534938 29967 consensus_queue.cc:1035] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "67f241431cf14e28981e6cafdd2e1e0c" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 35289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:06:37.544703 29972 mvcc.cc:205] Tried to move back new op lower bound from 7315603445891620864 to 7315603445716451328. Current Snapshot: MvccSnapshot[applied={T|T < 7315603445891620864}]
I20260806 17:06:37.551375 29468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:06:37.551580 29468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:06:37.554234 29468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3110a4e3cf24280a2833184fc6e4cc7: Sending DeleteTablet for 3 replicas of tablet 96bced8ce9fc4fb2b9661abf099cf3b9
I20260806 17:06:37.555143 29872 tablet_service.cc:1514] Processing DeleteTablet for tablet 96bced8ce9fc4fb2b9661abf099cf3b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:06:37 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:56054
I20260806 17:06:37.555192 29594 tablet_service.cc:1514] Processing DeleteTablet for tablet 96bced8ce9fc4fb2b9661abf099cf3b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:06:37 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:33486
I20260806 17:06:37.555212 29734 tablet_service.cc:1514] Processing DeleteTablet for tablet 96bced8ce9fc4fb2b9661abf099cf3b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:06:37 UTC) from {username='kudu', principal='kudu/127.9.111.126@KRBTEST.COM'} at 127.0.0.1:56922
I20260806 17:06:37.555752 29982 tablet_replica.cc:330] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c: stopping tablet replica
I20260806 17:06:37.556013 29983 tablet_replica.cc:330] T 96bced8ce9fc4fb2b9661abf099cf3b9 P d9b662ecb86945899c736e13e83874d1: stopping tablet replica
I20260806 17:06:37.556095 29982 raft_consensus.cc:2229] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:06:37.556305 29983 raft_consensus.cc:2229] T 96bced8ce9fc4fb2b9661abf099cf3b9 P d9b662ecb86945899c736e13e83874d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:06:37.556591 29982 raft_consensus.cc:2258] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:06:37.556718 29983 raft_consensus.cc:2258] T 96bced8ce9fc4fb2b9661abf099cf3b9 P d9b662ecb86945899c736e13e83874d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:06:37.556982 29984 tablet_replica.cc:330] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb: stopping tablet replica
I20260806 17:06:37.557049 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 29531
I20260806 17:06:37.557242 29984 raft_consensus.cc:2229] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:06:37.557540 29984 raft_consensus.cc:2258] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:06:37.559387 29982 ts_tablet_manager.cc:1905] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:06:37.561906 29984 ts_tablet_manager.cc:1905] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:06:37.561991 29982 ts_tablet_manager.cc:1918] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:06:37.562086 29982 log.cc:1196] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/96bced8ce9fc4fb2b9661abf099cf3b9
I20260806 17:06:37.562459 29982 ts_tablet_manager.cc:1939] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 67f241431cf14e28981e6cafdd2e1e0c: Deleting consensus metadata
I20260806 17:06:37.563800 29453 catalog_manager.cc:4841] TS 67f241431cf14e28981e6cafdd2e1e0c (127.9.111.67:35289): tablet 96bced8ce9fc4fb2b9661abf099cf3b9 (table test-table [id=7ec467dde5fa44e2a6079b7ce3c5ff6c]) successfully deleted
I20260806 17:06:37.564694 29984 ts_tablet_manager.cc:1918] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:06:37.564806 29984 log.cc:1196] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/96bced8ce9fc4fb2b9661abf099cf3b9
I20260806 17:06:37.565158 29984 ts_tablet_manager.cc:1939] T 96bced8ce9fc4fb2b9661abf099cf3b9 P 383c7678afb9409b923d077c40afe1bb: Deleting consensus metadata
I20260806 17:06:37.566284 29452 catalog_manager.cc:4841] TS 383c7678afb9409b923d077c40afe1bb (127.9.111.66:36769): tablet 96bced8ce9fc4fb2b9661abf099cf3b9 (table test-table [id=7ec467dde5fa44e2a6079b7ce3c5ff6c]) successfully deleted
W20260806 17:06:37.566811 29455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.111.65:46731 (error 108)
I20260806 17:06:37.567049 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 29670
I20260806 17:06:37.570473 29953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:37.570112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:46731 (local address 127.0.0.1:33500)
0806 17:06:37.570272 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:37.570363 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.65:46731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 17:06:37.570636 29455 catalog_manager.cc:4584] TS d9b662ecb86945899c736e13e83874d1 (127.9.111.65:46731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 96bced8ce9fc4fb2b9661abf099cf3b9: Network error: Client connection negotiation failed: client connection to 127.9.111.65:46731: connect: Connection refused (error 111)
I20260806 17:06:37.575057 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 29807
I20260806 17:06:37.581055 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 29436
2026-08-06T17:06:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2539 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T17:06:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:37Z Disabled control of system clock
I20260806 17:06:37.614632 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:42165
--webserver_interface=127.9.111.126
--webserver_port=0
--builtin_ntp_servers=127.9.111.84:40485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:42165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 17:06:37.729321 29989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:37.729630 29989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:37.729668 29989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:37.733894 29989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:37.733958 29989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:37.733973 29989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:37.734021 29989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:37.734057 29989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:37.739434 29989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:40485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:42165
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:42165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:37.740808 29989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:37.742316 29989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:37.751077 29994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:37.751083 29997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:37.751165 29989 server_base.cc:1034] running on GCE node
W20260806 17:06:37.751077 29995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:37.751963 29989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:37.753116 29989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:37.754319 29989 hybrid_clock.cc:648] HybridClock initialized: now 1786035997754290 us; error 41 us; skew 500 ppm
I20260806 17:06:37.756366 29989 webserver.cc:460] Webserver started at http://127.9.111.126:34027/ using document root <none> and password file <none>
I20260806 17:06:37.757066 29989 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:37.757138 29989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:37.757378 29989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:37.759632 29989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "bb9c9e61cd694c439fd9ff50b527415c"
format_stamp: "Formatted at 2026-08-06 17:06:37 on dist-test-slave-69rg"
server_key: "f2ebb89ff0bf1ad74e7df0b76d9a2971"
server_key_iv: "66b98f0eca7dc76c3759bc33d22dea4b"
server_key_version: "encryptionkey@0"
I20260806 17:06:37.760244 29989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "bb9c9e61cd694c439fd9ff50b527415c"
format_stamp: "Formatted at 2026-08-06 17:06:37 on dist-test-slave-69rg"
server_key: "f2ebb89ff0bf1ad74e7df0b76d9a2971"
server_key_iv: "66b98f0eca7dc76c3759bc33d22dea4b"
server_key_version: "encryptionkey@0"
I20260806 17:06:37.764058 29989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 17:06:37.766620 30003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:37.767828 29989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 17:06:37.767975 29989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "bb9c9e61cd694c439fd9ff50b527415c"
format_stamp: "Formatted at 2026-08-06 17:06:37 on dist-test-slave-69rg"
server_key: "f2ebb89ff0bf1ad74e7df0b76d9a2971"
server_key_iv: "66b98f0eca7dc76c3759bc33d22dea4b"
server_key_version: "encryptionkey@0"
I20260806 17:06:37.768092 29989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:37.794793 29989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:37.795559 29989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:37.795785 29989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:37.803498 29989 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:42165
I20260806 17:06:37.803508 30056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:42165 every 8 connection(s)
I20260806 17:06:37.804520 29989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:37.807739 30057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:37.811607 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 29989
I20260806 17:06:37.811792 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:37.812083 9661 external_mini_cluster.cc:1428] Setting key d8c192b5da9530fd6457da9d47b0035b
I20260806 17:06:37.813436 30057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c: Bootstrap starting.
I20260806 17:06:37.816171 30057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:37.817070 30057 log.cc:824] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:37.819110 30057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c: No bootstrap required, opened a new log
I20260806 17:06:37.822496 30060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:37.813479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.9.111.126:42165)
0806 17:06:37.814232 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:37.814254 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:37.814297 (+ 43us) server_negotiation.cc:407] Connection header received
0806 17:06:37.815168 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:37.815190 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:37.815666 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:37.816026 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:37.816505 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.817862 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.818758 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:37.819427 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:37.819598 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:37.820418 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:37.820426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:37.820442 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:37.820930 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:37.820999 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:37.821026 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:37.821120 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:37.821249 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 17:06:37.821514 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":131,"threads_started":1}
I20260806 17:06:37.823464 30057 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb9c9e61cd694c439fd9ff50b527415c" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42165 } }
I20260806 17:06:37.823722 30057 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:37.823791 30057 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb9c9e61cd694c439fd9ff50b527415c, State: Initialized, Role: FOLLOWER
I20260806 17:06:37.824227 30057 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb9c9e61cd694c439fd9ff50b527415c" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42165 } }
I20260806 17:06:37.824385 30057 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:37.824450 30057 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:37.824555 30057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:37.825436 30057 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb9c9e61cd694c439fd9ff50b527415c" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42165 } }
I20260806 17:06:37.825757 30057 leader_election.cc:302] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb9c9e61cd694c439fd9ff50b527415c; no voters:
I20260806 17:06:37.826108 30057 leader_election.cc:288] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:37.826223 30062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:37.826429 30062 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c [term 1 LEADER]: Becoming Leader. State: Replica: bb9c9e61cd694c439fd9ff50b527415c, State: Running, Role: LEADER
I20260806 17:06:37.826762 30062 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb9c9e61cd694c439fd9ff50b527415c" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42165 } }
I20260806 17:06:37.827317 30057 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:37.828310 30064 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb9c9e61cd694c439fd9ff50b527415c. Latest consensus state: current_term: 1 leader_uuid: "bb9c9e61cd694c439fd9ff50b527415c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb9c9e61cd694c439fd9ff50b527415c" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42165 } } }
I20260806 17:06:37.828452 30064 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:37.828824 30071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:37.828830 30063 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb9c9e61cd694c439fd9ff50b527415c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb9c9e61cd694c439fd9ff50b527415c" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42165 } } }
I20260806 17:06:37.828966 30063 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:37.831950 30071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:37.837383 30071 catalog_manager.cc:1347] Generated new cluster ID: d07bad2c5fe144f1aa87054332ae65b8
I20260806 17:06:37.837471 30071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:37.860785 30071 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:37.862150 30071 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:37.872845 30071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c: Generated new TSK 0
I20260806 17:06:37.873689 30071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:06:37.888170 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.65:0
--local_ip_for_outbound_sockets=127.9.111.65
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:42165
--builtin_ntp_servers=127.9.111.84:40485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 17:06:38.015442 30081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:38.015741 30081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:38.015782 30081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:38.020115 30081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:38.020200 30081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:38.020306 30081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.65
I20260806 17:06:38.025485 30081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:40485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.111.65
--webserver_port=0
--tserver_master_addrs=127.9.111.126:42165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.65
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:38.026872 30081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:38.028470 30081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:38.038312 30089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:38.038371 30086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:38.038363 30087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:38.038650 30081 server_base.cc:1034] running on GCE node
I20260806 17:06:38.039077 30081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:38.039698 30081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:38.040890 30081 hybrid_clock.cc:648] HybridClock initialized: now 1786035998040874 us; error 27 us; skew 500 ppm
I20260806 17:06:38.042837 30081 webserver.cc:460] Webserver started at http://127.9.111.65:38711/ using document root <none> and password file <none>
I20260806 17:06:38.043412 30081 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:38.043500 30081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:38.043735 30081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:38.045758 30081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-0/data/instance:
uuid: "183743363d1b4eeda7cc23f76b861c56"
format_stamp: "Formatted at 2026-08-06 17:06:38 on dist-test-slave-69rg"
server_key: "33ed9da373243ed21f2512170ae8367e"
server_key_iv: "9131a62824c3091e08d8cd418955215a"
server_key_version: "encryptionkey@0"
I20260806 17:06:38.046379 30081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance:
uuid: "183743363d1b4eeda7cc23f76b861c56"
format_stamp: "Formatted at 2026-08-06 17:06:38 on dist-test-slave-69rg"
server_key: "33ed9da373243ed21f2512170ae8367e"
server_key_iv: "9131a62824c3091e08d8cd418955215a"
server_key_version: "encryptionkey@0"
I20260806 17:06:38.050379 30081 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 17:06:38.052882 30095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:38.054210 30081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:06:38.054425 30081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-0/wal
uuid: "183743363d1b4eeda7cc23f76b861c56"
format_stamp: "Formatted at 2026-08-06 17:06:38 on dist-test-slave-69rg"
server_key: "33ed9da373243ed21f2512170ae8367e"
server_key_iv: "9131a62824c3091e08d8cd418955215a"
server_key_version: "encryptionkey@0"
I20260806 17:06:38.054555 30081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:38.069761 30081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:38.070533 30081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:38.070739 30081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:38.071403 30081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:38.072399 30081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:38.072464 30081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:38.072532 30081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:38.072579 30081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:38.081524 30081 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.65:41567
I20260806 17:06:38.081542 30212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.65:41567 every 8 connection(s)
I20260806 17:06:38.082499 30081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-0/data/info.pb
I20260806 17:06:38.086701 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 30081
I20260806 17:06:38.086895 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-0/wal/instance
I20260806 17:06:38.087270 9661 external_mini_cluster.cc:1428] Setting key 19c7b789590e14f8350f383d20c21c54
I20260806 17:06:38.089475 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.66:0
--local_ip_for_outbound_sockets=127.9.111.66
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:42165
--builtin_ntp_servers=127.9.111.84:40485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 17:06:38.093326 30060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.084489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:49689 (local address 127.9.111.126:42165)
0806 17:06:38.084660 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:38.084664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:38.085767 (+ 1103us) server_negotiation.cc:407] Connection header received
0806 17:06:38.087557 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:38.087561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:38.087624 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:38.087732 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:38.088933 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.089567 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.090850 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.091110 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.091223 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.092690 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:38.092696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:38.092702 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:38.092720 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:38.092747 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:38.092761 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:38.092839 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:38.092983 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 17:06:38.093058 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 17:06:38.094110 30215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.084886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:42165 (local address 127.9.111.65:49689)
0806 17:06:38.085612 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:38.085655 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:38.087321 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:38.087916 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:38.087924 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:38.088021 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:38.088766 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.088789 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.089739 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.089741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.090713 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.090723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.091416 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.091423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.091645 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.092364 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:06:38.092366 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:38.092422 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:06:38.092508 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:38.092854 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:38.092862 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:38.092866 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 17:06:38.092955 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 17:06:38.093169 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":132,"threads_started":1}
I20260806 17:06:38.095352 30213 heartbeater.cc:347] Connected to a master server at 127.9.111.126:42165
I20260806 17:06:38.095577 30213 heartbeater.cc:464] Registering TS with master...
I20260806 17:06:38.096083 30213 heartbeater.cc:511] Master 127.9.111.126:42165 requested a full tablet report, sending...
I20260806 17:06:38.097527 30021 ts_manager.cc:194] Registered new tserver with Master: 183743363d1b4eeda7cc23f76b861c56 (127.9.111.65:41567)
I20260806 17:06:38.098718 30021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.111.65:49689
W20260806 17:06:38.204172 30216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:38.204450 30216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:38.204487 30216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:38.208601 30216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:38.208668 30216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:38.208724 30216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.66
I20260806 17:06:38.213454 30216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:40485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.111.66
--webserver_port=0
--tserver_master_addrs=127.9.111.126:42165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.66
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:38.214589 30216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:38.215997 30216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:38.225756 30221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:38.225764 30222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:38.225957 30216 server_base.cc:1034] running on GCE node
W20260806 17:06:38.226146 30225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:38.226615 30216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:38.227339 30216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:38.228497 30216 hybrid_clock.cc:648] HybridClock initialized: now 1786035998228478 us; error 31 us; skew 500 ppm
I20260806 17:06:38.230687 30216 webserver.cc:460] Webserver started at http://127.9.111.66:37495/ using document root <none> and password file <none>
I20260806 17:06:38.231225 30216 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:38.231292 30216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:38.231463 30216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:38.233529 30216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-1/data/instance:
uuid: "3587824a26a14b30be8ab2b72a2f2fa6"
format_stamp: "Formatted at 2026-08-06 17:06:38 on dist-test-slave-69rg"
server_key: "b799a181e432fcd08749744a48fdca85"
server_key_iv: "f271a5ef2ef93970717a547b6d5a9674"
server_key_version: "encryptionkey@0"
I20260806 17:06:38.234122 30216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance:
uuid: "3587824a26a14b30be8ab2b72a2f2fa6"
format_stamp: "Formatted at 2026-08-06 17:06:38 on dist-test-slave-69rg"
server_key: "b799a181e432fcd08749744a48fdca85"
server_key_iv: "f271a5ef2ef93970717a547b6d5a9674"
server_key_version: "encryptionkey@0"
I20260806 17:06:38.237901 30216 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260806 17:06:38.240321 30231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:38.241745 30216 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 17:06:38.241947 30216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-1/wal
uuid: "3587824a26a14b30be8ab2b72a2f2fa6"
format_stamp: "Formatted at 2026-08-06 17:06:38 on dist-test-slave-69rg"
server_key: "b799a181e432fcd08749744a48fdca85"
server_key_iv: "f271a5ef2ef93970717a547b6d5a9674"
server_key_version: "encryptionkey@0"
I20260806 17:06:38.242065 30216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:38.254439 30216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:38.255122 30216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:38.255323 30216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:38.255916 30216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:38.256826 30216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:38.256893 30216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:38.256960 30216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:38.257009 30216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:38.265748 30216 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.66:44113
I20260806 17:06:38.265882 30348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.66:44113 every 8 connection(s)
I20260806 17:06:38.266755 30216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-1/data/info.pb
I20260806 17:06:38.267465 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 30216
I20260806 17:06:38.267616 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-1/wal/instance
I20260806 17:06:38.267916 9661 external_mini_cluster.cc:1428] Setting key 9db38babce18d6faad635e6062d7e0af
I20260806 17:06:38.270159 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.111.67:0
--local_ip_for_outbound_sockets=127.9.111.67
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:42165
--builtin_ntp_servers=127.9.111.84:40485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 17:06:38.277719 30060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.268732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.66:38941 (local address 127.9.111.126:42165)
0806 17:06:38.268854 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:38.268859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:38.270101 (+ 1242us) server_negotiation.cc:407] Connection header received
0806 17:06:38.271895 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:38.271898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:38.271981 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:38.272079 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:38.273038 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.273911 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.275303 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.275746 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.275878 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.277079 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:38.277086 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:38.277091 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:38.277112 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:38.277140 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:38.277153 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:38.277236 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:38.277406 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 17:06:38.277485 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 17:06:38.278648 30351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.269186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:42165 (local address 127.9.111.66:38941)
0806 17:06:38.269958 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:38.270009 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:38.271603 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:38.272276 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:38.272283 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:38.272372 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:38.272886 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.272899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.274047 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.274050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.275162 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.275173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.275914 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.275918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.276072 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.276796 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:06:38.276798 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:38.276840 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:06:38.276908 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:38.277255 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:38.277263 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:38.277268 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 17:06:38.277370 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 17:06:38.277602 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":146,"threads_started":1}
I20260806 17:06:38.279971 30349 heartbeater.cc:347] Connected to a master server at 127.9.111.126:42165
I20260806 17:06:38.280248 30349 heartbeater.cc:464] Registering TS with master...
I20260806 17:06:38.280963 30349 heartbeater.cc:511] Master 127.9.111.126:42165 requested a full tablet report, sending...
I20260806 17:06:38.282199 30021 ts_manager.cc:194] Registered new tserver with Master: 3587824a26a14b30be8ab2b72a2f2fa6 (127.9.111.66:44113)
I20260806 17:06:38.282790 30021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.111.66:38941
W20260806 17:06:38.385082 30352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:38.385349 30352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:38.385385 30352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:38.389389 30352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:38.389457 30352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:38.389549 30352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.111.67
I20260806 17:06:38.394493 30352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:40485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.111.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.111.67
--webserver_port=0
--tserver_master_addrs=127.9.111.126:42165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.111.67
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:38.395656 30352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:38.397068 30352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:38.406973 30357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:38.406989 30352 server_base.cc:1034] running on GCE node
W20260806 17:06:38.406982 30358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:38.407196 30360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:38.407618 30352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:38.408288 30352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:38.409448 30352 hybrid_clock.cc:648] HybridClock initialized: now 1786035998409419 us; error 43 us; skew 500 ppm
I20260806 17:06:38.411437 30352 webserver.cc:460] Webserver started at http://127.9.111.67:39379/ using document root <none> and password file <none>
I20260806 17:06:38.411999 30352 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:38.412063 30352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:38.412235 30352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:38.414465 30352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-2/data/instance:
uuid: "7f9dc7e432d94c8d8cdb2f8c767ff8a6"
format_stamp: "Formatted at 2026-08-06 17:06:38 on dist-test-slave-69rg"
server_key: "8393af0aad7e1d0859e0c906ed3f7490"
server_key_iv: "1ca92bd445a7327941f40b408f00904f"
server_key_version: "encryptionkey@0"
I20260806 17:06:38.415022 30352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance:
uuid: "7f9dc7e432d94c8d8cdb2f8c767ff8a6"
format_stamp: "Formatted at 2026-08-06 17:06:38 on dist-test-slave-69rg"
server_key: "8393af0aad7e1d0859e0c906ed3f7490"
server_key_iv: "1ca92bd445a7327941f40b408f00904f"
server_key_version: "encryptionkey@0"
I20260806 17:06:38.418910 30352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:06:38.421347 30366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:38.422703 30352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:06:38.422807 30352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-2/wal
uuid: "7f9dc7e432d94c8d8cdb2f8c767ff8a6"
format_stamp: "Formatted at 2026-08-06 17:06:38 on dist-test-slave-69rg"
server_key: "8393af0aad7e1d0859e0c906ed3f7490"
server_key_iv: "1ca92bd445a7327941f40b408f00904f"
server_key_version: "encryptionkey@0"
I20260806 17:06:38.422901 30352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:38.448257 30352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:38.448925 30352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:38.449082 30352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:38.449640 30352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 17:06:38.450619 30352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 17:06:38.450670 30352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:38.450711 30352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 17:06:38.450732 30352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:38.460980 30352 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.67:37893
I20260806 17:06:38.461051 30482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.67:37893 every 8 connection(s)
I20260806 17:06:38.462013 30352 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-2/data/info.pb
I20260806 17:06:38.468166 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 30352
I20260806 17:06:38.468281 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-2/wal/instance
I20260806 17:06:38.468559 9661 external_mini_cluster.cc:1428] Setting key a9b985208754372273cae32cc7155eba
I20260806 17:06:38.471691 30060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.463903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:34861 (local address 127.9.111.126:42165)
0806 17:06:38.464066 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:38.464070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:38.465092 (+ 1022us) server_negotiation.cc:407] Connection header received
0806 17:06:38.466617 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:38.466620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:38.466669 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:38.466750 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:38.467671 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.468223 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.469680 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.470002 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.470110 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.471265 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:38.471269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:38.471277 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:38.471289 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:38.471309 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:38.471318 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:38.471376 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:38.471527 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 17:06:38.471579 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 17:06:38.472668 30485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.464308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:42165 (local address 127.9.111.67:34861)
0806 17:06:38.464959 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:38.465001 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:38.466391 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:38.466908 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:38.466914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:38.466994 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:06:38.467515 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.467528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.468422 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.468424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.469561 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.469571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.470143 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.470146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.470297 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.470962 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 17:06:38.470964 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:06:38.471019 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 17:06:38.471099 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:06:38.471399 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:06:38.471407 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:06:38.471411 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 17:06:38.471509 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 17:06:38.471804 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":118,"threads_started":1}
I20260806 17:06:38.473752 30483 heartbeater.cc:347] Connected to a master server at 127.9.111.126:42165
I20260806 17:06:38.474030 30483 heartbeater.cc:464] Registering TS with master...
I20260806 17:06:38.474591 30483 heartbeater.cc:511] Master 127.9.111.126:42165 requested a full tablet report, sending...
I20260806 17:06:38.475585 30021 ts_manager.cc:194] Registered new tserver with Master: 7f9dc7e432d94c8d8cdb2f8c767ff8a6 (127.9.111.67:37893)
I20260806 17:06:38.476141 30021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.111.67:34861
I20260806 17:06:38.482666 9661 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 17:06:38.488461 30060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.484761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44790 (local address 127.9.111.126:42165)
0806 17:06:38.484907 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:38.484911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:38.484993 (+ 82us) server_negotiation.cc:407] Connection header received
0806 17:06:38.485103 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:38.485105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:38.485142 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:38.485211 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:38.485465 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.486003 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.486884 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.487090 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.487166 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.487951 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:38.487954 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:38.487956 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:38.487967 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:38.487997 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:38.488005 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:38.488071 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:38.488210 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 17:06:38.488264 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 17:06:38.491240 30021 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 17:06:38.494349 30021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 17:06:38.508534 30498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.503195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39854 (local address 127.9.111.65:41567)
0806 17:06:38.503806 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:38.503816 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:38.503838 (+ 22us) server_negotiation.cc:407] Connection header received
0806 17:06:38.503910 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:38.503915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:38.504084 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:38.504475 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:38.504603 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.505743 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.507338 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.507873 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.507999 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.508208 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:38.508274 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 17:06:38.508367 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":152,"threads_started":1}
I20260806 17:06:38.508528 30497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.502554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:41567 (local address 127.0.0.1:39854)
0806 17:06:38.503470 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:38.503483 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:38.503562 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:38.504224 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:38.504228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:38.504243 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:38.504345 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.504354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.505873 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.505877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.507181 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.507193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.507998 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.508002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.508107 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.508144 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 17:06:38.508245 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 17:06:38.508365 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":81,"thread_start_us":105,"threads_started":1}
I20260806 17:06:38.508791 30494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.502456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:44113 (local address 127.0.0.1:50086)
0806 17:06:38.503038 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:38.503088 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:38.503214 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:38.504054 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:38.504061 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:38.504084 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:38.504261 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.504268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.505938 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.505941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.507143 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.507153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.508215 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.508219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.508315 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.508339 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 17:06:38.508393 (+ 54us) client_negotiation.cc:240] Negotiation successful
0806 17:06:38.508468 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":99,"thread_start_us":90,"threads_started":1}
I20260806 17:06:38.508812 30495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.503132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:37893 (local address 127.0.0.1:53078)
0806 17:06:38.503682 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:38.503696 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:38.503777 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:38.504571 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:38.504574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:38.504591 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:38.504685 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.504694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.505612 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.505618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.507186 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.507193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.508445 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.508448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.508523 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.508550 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 17:06:38.508617 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 17:06:38.508679 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":118,"thread_start_us":97,"threads_started":1}
I20260806 17:06:38.509111 30496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.502575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50086 (local address 127.9.111.66:44113)
0806 17:06:38.503518 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:38.503532 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:38.503558 (+ 26us) server_negotiation.cc:407] Connection header received
0806 17:06:38.503640 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:38.503645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:38.503882 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:38.504423 (+ 541us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:38.504569 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.505750 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.507263 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.508072 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.508618 (+ 546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.508809 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:38.508864 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 17:06:38.508961 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":97,"threads_started":1}
I20260806 17:06:38.510071 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.503338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.9.111.67:37893)
0806 17:06:38.504001 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:38.504019 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:38.504044 (+ 25us) server_negotiation.cc:407] Connection header received
0806 17:06:38.504317 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:38.504321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:38.504446 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:38.504592 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:38.504782 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.505502 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.507341 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.508072 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.508192 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.508384 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:38.508968 (+ 584us) server_negotiation.cc:299] Negotiation successful
0806 17:06:38.509063 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":86,"threads_started":1}
I20260806 17:06:38.512055 30145 tablet_service.cc:1467] Processing CreateTablet for tablet f20464158b134fa8b016fa3b0324dd3b (DEFAULT_TABLE table=test-table [id=046e5aad513144a3a0fd29d1c6b0a925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:06:38.512127 30283 tablet_service.cc:1467] Processing CreateTablet for tablet f20464158b134fa8b016fa3b0324dd3b (DEFAULT_TABLE table=test-table [id=046e5aad513144a3a0fd29d1c6b0a925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:06:38.512470 30416 tablet_service.cc:1467] Processing CreateTablet for tablet f20464158b134fa8b016fa3b0324dd3b (DEFAULT_TABLE table=test-table [id=046e5aad513144a3a0fd29d1c6b0a925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 17:06:38.513120 30145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20464158b134fa8b016fa3b0324dd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:38.513156 30283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20464158b134fa8b016fa3b0324dd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:38.513372 30416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20464158b134fa8b016fa3b0324dd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:38.519820 30500 tablet_bootstrap.cc:492] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56: Bootstrap starting.
I20260806 17:06:38.521517 30502 tablet_bootstrap.cc:492] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6: Bootstrap starting.
I20260806 17:06:38.522116 30501 tablet_bootstrap.cc:492] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6: Bootstrap starting.
I20260806 17:06:38.522404 30500 tablet_bootstrap.cc:654] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:38.523360 30500 log.cc:824] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:38.523813 30502 tablet_bootstrap.cc:654] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:38.524005 30501 tablet_bootstrap.cc:654] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:38.524682 30502 log.cc:824] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:38.524803 30501 log.cc:824] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:38.526069 30500 tablet_bootstrap.cc:492] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56: No bootstrap required, opened a new log
I20260806 17:06:38.526279 30500 ts_tablet_manager.cc:1397] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 17:06:38.526738 30502 tablet_bootstrap.cc:492] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6: No bootstrap required, opened a new log
I20260806 17:06:38.526762 30501 tablet_bootstrap.cc:492] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6: No bootstrap required, opened a new log
I20260806 17:06:38.526947 30501 ts_tablet_manager.cc:1397] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 17:06:38.526947 30502 ts_tablet_manager.cc:1397] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 17:06:38.530257 30500 raft_consensus.cc:348] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587824a26a14b30be8ab2b72a2f2fa6" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44113 } } peers { permanent_uuid: "183743363d1b4eeda7cc23f76b861c56" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41567 } } peers { permanent_uuid: "7f9dc7e432d94c8d8cdb2f8c767ff8a6" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37893 } }
I20260806 17:06:38.530520 30500 raft_consensus.cc:374] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:38.530576 30500 raft_consensus.cc:729] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183743363d1b4eeda7cc23f76b861c56, State: Initialized, Role: FOLLOWER
I20260806 17:06:38.531050 30500 consensus_queue.cc:260] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587824a26a14b30be8ab2b72a2f2fa6" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44113 } } peers { permanent_uuid: "183743363d1b4eeda7cc23f76b861c56" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41567 } } peers { permanent_uuid: "7f9dc7e432d94c8d8cdb2f8c767ff8a6" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37893 } }
I20260806 17:06:38.531329 30501 raft_consensus.cc:348] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587824a26a14b30be8ab2b72a2f2fa6" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44113 } } peers { permanent_uuid: "183743363d1b4eeda7cc23f76b861c56" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41567 } } peers { permanent_uuid: "7f9dc7e432d94c8d8cdb2f8c767ff8a6" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37893 } }
I20260806 17:06:38.531354 30502 raft_consensus.cc:348] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587824a26a14b30be8ab2b72a2f2fa6" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44113 } } peers { permanent_uuid: "183743363d1b4eeda7cc23f76b861c56" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41567 } } peers { permanent_uuid: "7f9dc7e432d94c8d8cdb2f8c767ff8a6" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37893 } }
I20260806 17:06:38.531576 30502 raft_consensus.cc:374] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:38.531576 30501 raft_consensus.cc:374] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:38.531626 30501 raft_consensus.cc:729] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f9dc7e432d94c8d8cdb2f8c767ff8a6, State: Initialized, Role: FOLLOWER
I20260806 17:06:38.531636 30502 raft_consensus.cc:729] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3587824a26a14b30be8ab2b72a2f2fa6, State: Initialized, Role: FOLLOWER
I20260806 17:06:38.532053 30502 consensus_queue.cc:260] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587824a26a14b30be8ab2b72a2f2fa6" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44113 } } peers { permanent_uuid: "183743363d1b4eeda7cc23f76b861c56" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41567 } } peers { permanent_uuid: "7f9dc7e432d94c8d8cdb2f8c767ff8a6" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37893 } }
I20260806 17:06:38.532053 30501 consensus_queue.cc:260] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587824a26a14b30be8ab2b72a2f2fa6" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44113 } } peers { permanent_uuid: "183743363d1b4eeda7cc23f76b861c56" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41567 } } peers { permanent_uuid: "7f9dc7e432d94c8d8cdb2f8c767ff8a6" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37893 } }
I20260806 17:06:38.532166 30500 ts_tablet_manager.cc:1428] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 17:06:38.532549 30213 heartbeater.cc:503] Master 127.9.111.126:42165 was elected leader, sending a full tablet report...
I20260806 17:06:38.532992 30501 ts_tablet_manager.cc:1428] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 17:06:38.532992 30502 ts_tablet_manager.cc:1428] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 17:06:38.533162 30349 heartbeater.cc:503] Master 127.9.111.126:42165 was elected leader, sending a full tablet report...
I20260806 17:06:38.534080 30483 heartbeater.cc:503] Master 127.9.111.126:42165 was elected leader, sending a full tablet report...
W20260806 17:06:38.583763 30214 tablet.cc:2369] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 17:06:38.590546 30506 raft_consensus.cc:482] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:06:38.590788 30506 raft_consensus.cc:504] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587824a26a14b30be8ab2b72a2f2fa6" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44113 } } peers { permanent_uuid: "183743363d1b4eeda7cc23f76b861c56" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41567 } } peers { permanent_uuid: "7f9dc7e432d94c8d8cdb2f8c767ff8a6" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37893 } }
I20260806 17:06:38.591989 30506 leader_election.cc:288] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3587824a26a14b30be8ab2b72a2f2fa6 (127.9.111.66:44113), 7f9dc7e432d94c8d8cdb2f8c767ff8a6 (127.9.111.67:37893)
I20260806 17:06:38.597122 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.592758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:36563 (local address 127.9.111.67:37893)
0806 17:06:38.592903 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:38.592908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:38.593027 (+ 119us) server_negotiation.cc:407] Connection header received
0806 17:06:38.593128 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:38.593130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:38.593168 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:38.593259 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:38.593495 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.594228 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.595714 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.596357 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.596493 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.596545 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:38.596903 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 17:06:38.596986 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 17:06:38.597122 30496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.592423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.65:54585 (local address 127.9.111.66:44113)
0806 17:06:38.592545 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:38.592548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:38.592580 (+ 32us) server_negotiation.cc:407] Connection header received
0806 17:06:38.592685 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:38.592688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:38.592744 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:38.592839 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:38.593250 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.594076 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.595714 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.596357 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.596493 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.596545 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:38.596903 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 17:06:38.596987 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 17:06:38.597143 30215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.592324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:44113 (local address 127.9.111.65:54585)
0806 17:06:38.592464 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:38.592482 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:38.592581 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:38.592956 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:38.592960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:38.592985 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:38.593121 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.593130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.594190 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.594193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.595557 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.595569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.596518 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.596522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.596712 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.596762 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 17:06:38.596851 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 17:06:38.596972 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":103}
I20260806 17:06:38.597520 30508 raft_consensus.cc:482] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 17:06:38.597648 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.592633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.67:37893 (local address 127.9.111.65:36563)
0806 17:06:38.592942 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:38.592955 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:38.593036 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:38.593280 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:38.593282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:38.593299 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:38.593388 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.593392 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.594323 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.594326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.595557 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.595569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.596519 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.596522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.596689 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.596762 (+ 73us) client_negotiation.cc:769] Sending connection context
0806 17:06:38.596851 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 17:06:38.596972 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":141,"thread_start_us":130,"threads_started":1}
I20260806 17:06:38.597905 30508 raft_consensus.cc:504] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587824a26a14b30be8ab2b72a2f2fa6" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44113 } } peers { permanent_uuid: "183743363d1b4eeda7cc23f76b861c56" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41567 } } peers { permanent_uuid: "7f9dc7e432d94c8d8cdb2f8c767ff8a6" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37893 } }
I20260806 17:06:38.597934 30303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20464158b134fa8b016fa3b0324dd3b" candidate_uuid: "183743363d1b4eeda7cc23f76b861c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3587824a26a14b30be8ab2b72a2f2fa6" is_pre_election: true
I20260806 17:06:38.598281 30437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20464158b134fa8b016fa3b0324dd3b" candidate_uuid: "183743363d1b4eeda7cc23f76b861c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f9dc7e432d94c8d8cdb2f8c767ff8a6" is_pre_election: true
I20260806 17:06:38.598263 30303 raft_consensus.cc:2454] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 183743363d1b4eeda7cc23f76b861c56 in term 0.
I20260806 17:06:38.598456 30437 raft_consensus.cc:2454] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 183743363d1b4eeda7cc23f76b861c56 in term 0.
I20260806 17:06:38.598846 30098 leader_election.cc:302] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 183743363d1b4eeda7cc23f76b861c56, 3587824a26a14b30be8ab2b72a2f2fa6; no voters:
I20260806 17:06:38.599387 30506 raft_consensus.cc:2781] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 17:06:38.599485 30506 raft_consensus.cc:482] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 17:06:38.599539 30506 raft_consensus.cc:3037] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:38.599520 30508 leader_election.cc:288] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3587824a26a14b30be8ab2b72a2f2fa6 (127.9.111.66:44113), 183743363d1b4eeda7cc23f76b861c56 (127.9.111.65:41567)
I20260806 17:06:38.600980 30506 raft_consensus.cc:504] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587824a26a14b30be8ab2b72a2f2fa6" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44113 } } peers { permanent_uuid: "183743363d1b4eeda7cc23f76b861c56" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41567 } } peers { permanent_uuid: "7f9dc7e432d94c8d8cdb2f8c767ff8a6" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37893 } }
I20260806 17:06:38.601373 30506 leader_election.cc:288] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [CANDIDATE]: Term 1 election: Requested vote from peers 3587824a26a14b30be8ab2b72a2f2fa6 (127.9.111.66:44113), 7f9dc7e432d94c8d8cdb2f8c767ff8a6 (127.9.111.67:37893)
I20260806 17:06:38.601704 30303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20464158b134fa8b016fa3b0324dd3b" candidate_uuid: "183743363d1b4eeda7cc23f76b861c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3587824a26a14b30be8ab2b72a2f2fa6"
I20260806 17:06:38.601704 30437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20464158b134fa8b016fa3b0324dd3b" candidate_uuid: "183743363d1b4eeda7cc23f76b861c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f9dc7e432d94c8d8cdb2f8c767ff8a6"
I20260806 17:06:38.601859 30437 raft_consensus.cc:3037] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:38.601833 30303 raft_consensus.cc:3037] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:38.603153 30303 raft_consensus.cc:2454] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 183743363d1b4eeda7cc23f76b861c56 in term 1.
I20260806 17:06:38.603152 30437 raft_consensus.cc:2454] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 183743363d1b4eeda7cc23f76b861c56 in term 1.
I20260806 17:06:38.603468 30098 leader_election.cc:302] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 183743363d1b4eeda7cc23f76b861c56, 3587824a26a14b30be8ab2b72a2f2fa6; no voters:
I20260806 17:06:38.603646 30506 raft_consensus.cc:2781] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:38.603924 30506 raft_consensus.cc:686] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [term 1 LEADER]: Becoming Leader. State: Replica: 183743363d1b4eeda7cc23f76b861c56, State: Running, Role: LEADER
I20260806 17:06:38.604297 30506 consensus_queue.cc:237] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587824a26a14b30be8ab2b72a2f2fa6" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44113 } } peers { permanent_uuid: "183743363d1b4eeda7cc23f76b861c56" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41567 } } peers { permanent_uuid: "7f9dc7e432d94c8d8cdb2f8c767ff8a6" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37893 } }
I20260806 17:06:38.604401 30496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.600038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:33325 (local address 127.9.111.66:44113)
0806 17:06:38.600143 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:38.600147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:38.600538 (+ 391us) server_negotiation.cc:407] Connection header received
0806 17:06:38.600692 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:38.600695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:38.600738 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:38.600785 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:38.601169 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.601923 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.603069 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.603640 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.603756 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.603833 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:38.604199 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 17:06:38.604281 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 17:06:38.604760 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.599735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.66:44113 (local address 127.9.111.67:33325)
0806 17:06:38.600441 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:38.600455 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:38.600551 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:38.600907 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:38.600910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:38.600925 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:38.601040 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.601045 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.602034 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.602036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.602928 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.602939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.603950 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.603953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.604049 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.604074 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 17:06:38.604570 (+ 496us) client_negotiation.cc:240] Negotiation successful
0806 17:06:38.604642 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":349,"thread_start_us":156,"threads_started":1}
I20260806 17:06:38.605324 30303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20464158b134fa8b016fa3b0324dd3b" candidate_uuid: "7f9dc7e432d94c8d8cdb2f8c767ff8a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3587824a26a14b30be8ab2b72a2f2fa6" is_pre_election: true
I20260806 17:06:38.605501 30303 raft_consensus.cc:2379] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7f9dc7e432d94c8d8cdb2f8c767ff8a6 in current term 1: Already voted for candidate 183743363d1b4eeda7cc23f76b861c56 in this term.
I20260806 17:06:38.606274 30498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.599971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.111.67:41231 (local address 127.9.111.65:41567)
0806 17:06:38.600154 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:38.600160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:38.600174 (+ 14us) server_negotiation.cc:407] Connection header received
0806 17:06:38.600215 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:38.600218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:38.600263 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:38.600392 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 17:06:38.600658 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.601421 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.604496 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.605051 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.605158 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.605200 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:38.606086 (+ 886us) server_negotiation.cc:299] Negotiation successful
0806 17:06:38.606177 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260806 17:06:38.606283 30485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.599736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:41567 (local address 127.9.111.67:41231)
0806 17:06:38.599929 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:38.599945 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:06:38.600021 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:06:38.600407 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:06:38.600410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:06:38.600427 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 17:06:38.600536 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.600542 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.601542 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.601545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.604351 (+ 2806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:06:38.604361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.605860 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.605864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:06:38.605964 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.605987 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 17:06:38.606064 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 17:06:38.606130 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260806 17:06:38.606842 30165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20464158b134fa8b016fa3b0324dd3b" candidate_uuid: "7f9dc7e432d94c8d8cdb2f8c767ff8a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "183743363d1b4eeda7cc23f76b861c56" is_pre_election: true
I20260806 17:06:38.607439 30370 leader_election.cc:302] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f9dc7e432d94c8d8cdb2f8c767ff8a6; no voters: 183743363d1b4eeda7cc23f76b861c56, 3587824a26a14b30be8ab2b72a2f2fa6
I20260806 17:06:38.607942 30508 raft_consensus.cc:2726] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 17:06:38.608291 30019 catalog_manager.cc:5495] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 183743363d1b4eeda7cc23f76b861c56 (127.9.111.65). New cstate: current_term: 1 leader_uuid: "183743363d1b4eeda7cc23f76b861c56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587824a26a14b30be8ab2b72a2f2fa6" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "183743363d1b4eeda7cc23f76b861c56" member_type: VOTER last_known_addr { host: "127.9.111.65" port: 41567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f9dc7e432d94c8d8cdb2f8c767ff8a6" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37893 } health_report { overall_health: UNKNOWN } } }
I20260806 17:06:38.631539 30498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.627885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39862 (local address 127.9.111.65:41567)
0806 17:06:38.628013 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:38.628017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:38.628029 (+ 12us) server_negotiation.cc:407] Connection header received
0806 17:06:38.628131 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:38.628134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:38.628242 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:38.628333 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:06:38.628595 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.629108 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.630078 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:38.630328 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:38.630440 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:38.630743 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:06:38.631147 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:06:38.631289 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:38.631325 (+ 36us) server_negotiation.cc:299] Negotiation successful
0806 17:06:38.631375 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 17:06:38.638167 30437 raft_consensus.cc:1261] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6 [term 1 FOLLOWER]: Refusing update from remote peer 183743363d1b4eeda7cc23f76b861c56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:06:38.638128 30303 raft_consensus.cc:1261] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6 [term 1 FOLLOWER]: Refusing update from remote peer 183743363d1b4eeda7cc23f76b861c56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:06:38.639060 30511 consensus_queue.cc:1035] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f9dc7e432d94c8d8cdb2f8c767ff8a6" member_type: VOTER last_known_addr { host: "127.9.111.67" port: 37893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:06:38.639248 30506 consensus_queue.cc:1035] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3587824a26a14b30be8ab2b72a2f2fa6" member_type: VOTER last_known_addr { host: "127.9.111.66" port: 44113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:06:38.648315 30516 mvcc.cc:205] Tried to move back new op lower bound from 7315603450413449216 to 7315603450287550464. Current Snapshot: MvccSnapshot[applied={T|T < 7315603450413449216}]
I20260806 17:06:38.656167 30021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:06:38.656338 30021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 17:06:38.658995 30021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb9c9e61cd694c439fd9ff50b527415c: Sending DeleteTablet for 3 replicas of tablet f20464158b134fa8b016fa3b0324dd3b
I20260806 17:06:38.659904 30416 tablet_service.cc:1514] Processing DeleteTablet for tablet f20464158b134fa8b016fa3b0324dd3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:06:38 UTC) from {username='slave'} at 127.0.0.1:53078
I20260806 17:06:38.659866 30145 tablet_service.cc:1514] Processing DeleteTablet for tablet f20464158b134fa8b016fa3b0324dd3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:06:38 UTC) from {username='slave'} at 127.0.0.1:39854
I20260806 17:06:38.660043 30283 tablet_service.cc:1514] Processing DeleteTablet for tablet f20464158b134fa8b016fa3b0324dd3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 17:06:38 UTC) from {username='slave'} at 127.0.0.1:50086
I20260806 17:06:38.660413 30526 tablet_replica.cc:330] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56: stopping tablet replica
I20260806 17:06:38.660591 30527 tablet_replica.cc:330] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6: stopping tablet replica
I20260806 17:06:38.660925 30526 raft_consensus.cc:2229] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [term 1 LEADER]: Raft consensus shutting down.
I20260806 17:06:38.660929 30527 raft_consensus.cc:2229] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:06:38.661124 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 30081
I20260806 17:06:38.661278 30527 raft_consensus.cc:2258] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:06:38.661484 30526 raft_consensus.cc:2258] T f20464158b134fa8b016fa3b0324dd3b P 183743363d1b4eeda7cc23f76b861c56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:06:38.662030 30525 tablet_replica.cc:330] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6: stopping tablet replica
I20260806 17:06:38.662388 30525 raft_consensus.cc:2229] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 17:06:38.662725 30525 raft_consensus.cc:2258] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 17:06:38.664459 30525 ts_tablet_manager.cc:1905] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:06:38.664484 30527 ts_tablet_manager.cc:1905] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 17:06:38.666844 30525 ts_tablet_manager.cc:1918] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:06:38.667053 30525 log.cc:1196] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-2/wal/wals/f20464158b134fa8b016fa3b0324dd3b
I20260806 17:06:38.667409 30525 ts_tablet_manager.cc:1939] T f20464158b134fa8b016fa3b0324dd3b P 7f9dc7e432d94c8d8cdb2f8c767ff8a6: Deleting consensus metadata
I20260806 17:06:38.667994 30527 ts_tablet_manager.cc:1918] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 17:06:38.668102 30527 log.cc:1196] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6: Deleting WAL directory at /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786035732873553-9661-0/minicluster-data/ts-1/wal/wals/f20464158b134fa8b016fa3b0324dd3b
I20260806 17:06:38.668423 30527 ts_tablet_manager.cc:1939] T f20464158b134fa8b016fa3b0324dd3b P 3587824a26a14b30be8ab2b72a2f2fa6: Deleting consensus metadata
I20260806 17:06:38.668699 30006 catalog_manager.cc:4841] TS 7f9dc7e432d94c8d8cdb2f8c767ff8a6 (127.9.111.67:37893): tablet f20464158b134fa8b016fa3b0324dd3b (table test-table [id=046e5aad513144a3a0fd29d1c6b0a925]) successfully deleted
I20260806 17:06:38.669912 30006 catalog_manager.cc:4841] TS 3587824a26a14b30be8ab2b72a2f2fa6 (127.9.111.66:44113): tablet f20464158b134fa8b016fa3b0324dd3b (table test-table [id=046e5aad513144a3a0fd29d1c6b0a925]) successfully deleted
W20260806 17:06:38.670444 30007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.111.65:41567 (error 108)
I20260806 17:06:38.670714 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 30216
I20260806 17:06:38.671779 30494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:38.671313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.65:41567 (local address 127.0.0.1:39866)
0806 17:06:38.671586 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 17:06:38.671677 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.65:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
W20260806 17:06:38.672058 30007 catalog_manager.cc:4584] TS 183743363d1b4eeda7cc23f76b861c56 (127.9.111.65:41567): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f20464158b134fa8b016fa3b0324dd3b: Network error: Client connection negotiation failed: client connection to 127.9.111.65:41567: connect: Connection refused (error 111)
I20260806 17:06:38.678634 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 30352
I20260806 17:06:38.685137 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 29989
2026-08-06T17:06:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1103 ms)
[----------] 6 tests from EncryptionPolicyTest (10919 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:06:38 dist-test-slave-69rg krb5kdc[30531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:06:38 dist-test-slave-69rg krb5kdc[30531](info): set up 2 sockets
Aug 06 17:06:38 dist-test-slave-69rg krb5kdc[30531](info): commencing operation
krb5kdc: starting...
W20260806 17:06:39.765900 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:06:39 dist-test-slave-69rg krb5kdc[30531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:06:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 17:06:39.902217 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39455
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:40.014921 30545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:40.015272 30545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:40.015318 30545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:40.019719 30545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:40.019815 30545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:40.019836 30545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:40.019857 30545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:40.019873 30545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:40.025768 30545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39455
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:40.027043 30545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:40.028159 30545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:40.037669 30553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:40.037652 30550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:40.037725 30545 server_base.cc:1034] running on GCE node
W20260806 17:06:40.037652 30551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:40.038605 30545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:40.039950 30545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:40.041193 30545 hybrid_clock.cc:648] HybridClock initialized: now 1786036000041158 us; error 58 us; skew 500 ppm
Aug 06 17:06:40 dist-test-slave-69rg krb5kdc[30531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:40.047909 30545 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 17:06:40.048898 30545 webserver.cc:460] Webserver started at http://127.0.0.1:37533/ using document root <none> and password file <none>
I20260806 17:06:40.049489 30545 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:40.049552 30545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:40.049721 30545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:40.051848 30545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "0f5ecee5af3443858901c1361dfa4fc4"
format_stamp: "Formatted at 2026-08-06 17:06:40 on dist-test-slave-69rg"
server_key: "32cf93fbbc5813ede38996923735e16b"
server_key_iv: "9402893cfb6dd237f4d7701da39e7a57"
server_key_version: "encryptionkey@0"
I20260806 17:06:40.052381 30545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "0f5ecee5af3443858901c1361dfa4fc4"
format_stamp: "Formatted at 2026-08-06 17:06:40 on dist-test-slave-69rg"
server_key: "32cf93fbbc5813ede38996923735e16b"
server_key_iv: "9402893cfb6dd237f4d7701da39e7a57"
server_key_version: "encryptionkey@0"
I20260806 17:06:40.056126 30545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 17:06:40.058670 30560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:40.060331 30545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:06:40.060523 30545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "0f5ecee5af3443858901c1361dfa4fc4"
format_stamp: "Formatted at 2026-08-06 17:06:40 on dist-test-slave-69rg"
server_key: "32cf93fbbc5813ede38996923735e16b"
server_key_iv: "9402893cfb6dd237f4d7701da39e7a57"
server_key_version: "encryptionkey@0"
I20260806 17:06:40.060678 30545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:40.096153 30545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:40.100065 30545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:40.100253 30545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:40.107736 30545 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39455
I20260806 17:06:40.107762 30612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39455 every 8 connection(s)
I20260806 17:06:40.108834 30545 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:40.110184 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 30545
I20260806 17:06:40.110303 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:40.110625 9661 external_mini_cluster.cc:1428] Setting key 18e5b9d1967239c7c9a3bcb81d1fcb41
I20260806 17:06:40.111980 30613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:40.118803 30613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4: Bootstrap starting.
Aug 06 17:06:40 dist-test-slave-69rg krb5kdc[30531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786035999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 17:06:40.121325 30613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:40.122154 30613 log.cc:824] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:40.124126 30613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4: No bootstrap required, opened a new log
I20260806 17:06:40.125526 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:40.112078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37485 (local address 127.0.0.1:39455)
0806 17:06:40.112537 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:40.112552 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:40.112588 (+ 36us) server_negotiation.cc:407] Connection header received
0806 17:06:40.113248 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:40.113264 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:40.113589 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:40.113939 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:40.114846 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:40.115843 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:40.117024 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:40.117489 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:40.117616 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:40.120532 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:40.120538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:40.120551 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:40.120580 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:40.122587 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:40.123136 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:40.123141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:40.123145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:40.123259 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:40.123610 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:40.123617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:40.123619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:40.124018 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:40.124250 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:40.124582 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 17:06:40.124819 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":172,"threads_started":1}
I20260806 17:06:40.128580 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:06:40.128511 30613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f5ecee5af3443858901c1361dfa4fc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39455 } }
I20260806 17:06:40.128819 30613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:40.128881 30613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5ecee5af3443858901c1361dfa4fc4, State: Initialized, Role: FOLLOWER
I20260806 17:06:40.129328 30613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f5ecee5af3443858901c1361dfa4fc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39455 } }
I20260806 17:06:40.129477 30613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:40.129542 30613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:40.129637 30613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:40.130767 30613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f5ecee5af3443858901c1361dfa4fc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39455 } }
I20260806 17:06:40.131202 30613 leader_election.cc:302] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f5ecee5af3443858901c1361dfa4fc4; no voters:
I20260806 17:06:40.131551 30613 leader_election.cc:288] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:40.131698 30623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:40.131891 30623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4 [term 1 LEADER]: Becoming Leader. State: Replica: 0f5ecee5af3443858901c1361dfa4fc4, State: Running, Role: LEADER
I20260806 17:06:40.132185 30623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f5ecee5af3443858901c1361dfa4fc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39455 } }
I20260806 17:06:40.132891 30613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:40.134236 30624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f5ecee5af3443858901c1361dfa4fc4. Latest consensus state: current_term: 1 leader_uuid: "0f5ecee5af3443858901c1361dfa4fc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f5ecee5af3443858901c1361dfa4fc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39455 } } }
I20260806 17:06:40.134380 30624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:40.134682 30625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f5ecee5af3443858901c1361dfa4fc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f5ecee5af3443858901c1361dfa4fc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39455 } } }
I20260806 17:06:40.134775 30625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:40.134780 30633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:40.138522 30633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:40.141323 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:40.130918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50109 (local address 127.0.0.1:39455)
0806 17:06:40.131037 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:40.131042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:40.131275 (+ 233us) server_negotiation.cc:407] Connection header received
0806 17:06:40.131454 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:40.131457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:40.131526 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:40.131601 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:40.132526 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:40.133513 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:40.135037 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:40.135425 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:40.135550 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:40.137354 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:40.137359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:40.137362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:40.137391 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:40.139197 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:40.139841 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:40.139844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:40.139847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:40.139937 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:40.140337 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:40.140341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:40.140342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:40.140575 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:40.140732 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:40.141057 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 17:06:40.141160 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 17:06:40.144564 30633 catalog_manager.cc:1347] Generated new cluster ID: d7c524ba413a497ea61a2ad4be5a1d59
I20260806 17:06:40.144642 30633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:40.157243 30633 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:40.158435 30633 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:40.172701 30633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f5ecee5af3443858901c1361dfa4fc4: Generated new TSK 0
I20260806 17:06:40.173638 30633 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:06:40.179267 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 30545
W20260806 17:06:40.186084 30529 connection.cc:535] client connection to 127.0.0.1:39455 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39455): Connection reset by peer (error 104)
2026-08-06T17:06:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1492 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:06:40 dist-test-slave-69rg krb5kdc[30644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:06:40 dist-test-slave-69rg krb5kdc[30644](info): set up 2 sockets
Aug 06 17:06:40 dist-test-slave-69rg krb5kdc[30644](info): commencing operation
krb5kdc: starting...
W20260806 17:06:41.241092 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:06:41 dist-test-slave-69rg krb5kdc[30644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:06:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 17:06:41.380483 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34221
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:41.494040 30659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:41.494323 30659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:41.494366 30659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:41.498318 30659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:41.498375 30659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:41.498394 30659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:41.498411 30659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:41.498426 30659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:41.503763 30659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34221
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:41.504835 30659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:41.505894 30659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:41.515128 30667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:41.515345 30659 server_base.cc:1034] running on GCE node
W20260806 17:06:41.515158 30665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:41.515226 30664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:41.516103 30659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:41.517359 30659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:41.518543 30659 hybrid_clock.cc:648] HybridClock initialized: now 1786036001518518 us; error 41 us; skew 500 ppm
Aug 06 17:06:41 dist-test-slave-69rg krb5kdc[30644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:41.521665 30659 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 17:06:41.522675 30659 webserver.cc:460] Webserver started at http://127.0.0.1:34459/ using document root <none> and password file <none>
I20260806 17:06:41.523278 30659 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:41.523348 30659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:41.523523 30659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:41.525590 30659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "6d29fff857514b5fac145b1b7bc7bd35"
format_stamp: "Formatted at 2026-08-06 17:06:41 on dist-test-slave-69rg"
server_key: "df4970e1e7256920c6aa77469778ff52"
server_key_iv: "93b9a6390ce7fc684ab4744096b98c4e"
server_key_version: "encryptionkey@0"
I20260806 17:06:41.526183 30659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "6d29fff857514b5fac145b1b7bc7bd35"
format_stamp: "Formatted at 2026-08-06 17:06:41 on dist-test-slave-69rg"
server_key: "df4970e1e7256920c6aa77469778ff52"
server_key_iv: "93b9a6390ce7fc684ab4744096b98c4e"
server_key_version: "encryptionkey@0"
I20260806 17:06:41.529954 30659 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 17:06:41.532598 30675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:41.534335 30659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:06:41.534513 30659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "6d29fff857514b5fac145b1b7bc7bd35"
format_stamp: "Formatted at 2026-08-06 17:06:41 on dist-test-slave-69rg"
server_key: "df4970e1e7256920c6aa77469778ff52"
server_key_iv: "93b9a6390ce7fc684ab4744096b98c4e"
server_key_version: "encryptionkey@0"
I20260806 17:06:41.534673 30659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:41.553525 30659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:41.556671 30659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:41.556895 30659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:41.564273 30659 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34221
I20260806 17:06:41.564385 30727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34221 every 8 connection(s)
I20260806 17:06:41.565286 30659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:41.567945 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 30659
I20260806 17:06:41.568065 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:41.568348 30728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:41.568398 9661 external_mini_cluster.cc:1428] Setting key f5635acbcd0f430aec805d6cbd52d578
I20260806 17:06:41.575699 30728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35: Bootstrap starting.
I20260806 17:06:41.578730 30728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:41.579217 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:41.569799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41867 (local address 127.0.0.1:34221)
0806 17:06:41.570359 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:41.570378 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:41.570424 (+ 46us) server_negotiation.cc:407] Connection header received
0806 17:06:41.571112 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:41.571143 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:41.571576 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:41.571953 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:41.572403 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:41.573915 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:41.574858 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:41.575575 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:41.575753 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:41.576897 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:41.576907 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:41.576926 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:41.577484 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:41.577559 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:41.577590 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:41.577729 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:41.578000 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 17:06:41.578164 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":170,"threads_started":1}
I20260806 17:06:41.579655 30728 log.cc:824] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:41.582023 30728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35: No bootstrap required, opened a new log
I20260806 17:06:41.582609 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:06:41.587066 30728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d29fff857514b5fac145b1b7bc7bd35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34221 } }
I20260806 17:06:41.587380 30728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:41.587441 30728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d29fff857514b5fac145b1b7bc7bd35, State: Initialized, Role: FOLLOWER
I20260806 17:06:41.587918 30728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d29fff857514b5fac145b1b7bc7bd35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34221 } }
I20260806 17:06:41.588083 30728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:41.588191 30728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:41.588308 30728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:41.589454 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:41.585107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38165 (local address 127.0.0.1:34221)
0806 17:06:41.585241 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:41.585246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:41.585512 (+ 266us) server_negotiation.cc:407] Connection header received
0806 17:06:41.585619 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:41.585622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:41.585672 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:41.585745 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:41.586064 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:41.586598 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:41.587747 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:41.587978 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:41.588060 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:41.588975 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:41.588978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:41.588981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:41.588992 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:41.589017 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:41.589027 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:41.589101 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:41.589249 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 17:06:41.589279 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 17:06:41.589701 30728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d29fff857514b5fac145b1b7bc7bd35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34221 } }
I20260806 17:06:41.590103 30728 leader_election.cc:302] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d29fff857514b5fac145b1b7bc7bd35; no voters:
I20260806 17:06:41.590505 30728 leader_election.cc:288] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:41.590634 30738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:41.590920 30738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35 [term 1 LEADER]: Becoming Leader. State: Replica: 6d29fff857514b5fac145b1b7bc7bd35, State: Running, Role: LEADER
I20260806 17:06:41.591280 30738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d29fff857514b5fac145b1b7bc7bd35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34221 } }
I20260806 17:06:41.591831 30728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:41.593009 30740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d29fff857514b5fac145b1b7bc7bd35. Latest consensus state: current_term: 1 leader_uuid: "6d29fff857514b5fac145b1b7bc7bd35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d29fff857514b5fac145b1b7bc7bd35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34221 } } }
I20260806 17:06:41.593170 30740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:41.593479 30739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d29fff857514b5fac145b1b7bc7bd35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d29fff857514b5fac145b1b7bc7bd35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34221 } } }
I20260806 17:06:41.593568 30747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:41.593667 30739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:41.597195 30747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:41.603533 30747 catalog_manager.cc:1347] Generated new cluster ID: f69c3458e1304aa1a799123eebcc5379
I20260806 17:06:41.603627 30747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:41.619489 30747 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:41.620808 30747 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:41.630087 30747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d29fff857514b5fac145b1b7bc7bd35: Generated new TSK 0
I20260806 17:06:41.630978 30747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:06:41.638973 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 30659
W20260806 17:06:41.646013 30642 connection.cc:535] client connection to 127.0.0.1:34221 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34221): Connection reset by peer (error 104)
2026-08-06T17:06:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:06:41 dist-test-slave-69rg krb5kdc[30759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:06:41 dist-test-slave-69rg krb5kdc[30759](info): set up 2 sockets
Aug 06 17:06:41 dist-test-slave-69rg krb5kdc[30759](info): commencing operation
krb5kdc: starting...
W20260806 17:06:42.717607 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:06:42 dist-test-slave-69rg krb5kdc[30759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:06:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 17:06:42.853282 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36407
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:42.967087 30772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:42.967450 30772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:42.967546 30772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:42.971823 30772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:42.971925 30772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:42.971966 30772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:42.972002 30772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:42.972039 30772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:42.977540 30772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36407
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:42.978850 30772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:42.979899 30772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:42.988330 30780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:42.988451 30778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:42.988451 30777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:42.988723 30772 server_base.cc:1034] running on GCE node
I20260806 17:06:42.989312 30772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:42.990550 30772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:42.991744 30772 hybrid_clock.cc:648] HybridClock initialized: now 1786036002991731 us; error 31 us; skew 500 ppm
Aug 06 17:06:42 dist-test-slave-69rg krb5kdc[30759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:42.994772 30772 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 17:06:42.995653 30772 webserver.cc:460] Webserver started at http://127.0.0.1:41695/ using document root <none> and password file <none>
I20260806 17:06:42.996173 30772 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:42.996232 30772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:42.996402 30772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:42.998528 30772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "abc41bb18e654b04aa9f8e4b6c446a94"
format_stamp: "Formatted at 2026-08-06 17:06:42 on dist-test-slave-69rg"
server_key: "b24e8196caea61491c308a63b205c4d5"
server_key_iv: "04a371a516ab3e4c48ffb058909ff54a"
server_key_version: "encryptionkey@0"
I20260806 17:06:42.999080 30772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "abc41bb18e654b04aa9f8e4b6c446a94"
format_stamp: "Formatted at 2026-08-06 17:06:42 on dist-test-slave-69rg"
server_key: "b24e8196caea61491c308a63b205c4d5"
server_key_iv: "04a371a516ab3e4c48ffb058909ff54a"
server_key_version: "encryptionkey@0"
I20260806 17:06:43.003065 30772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 17:06:43.005695 30787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:43.007051 30772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 17:06:43.007231 30772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "abc41bb18e654b04aa9f8e4b6c446a94"
format_stamp: "Formatted at 2026-08-06 17:06:42 on dist-test-slave-69rg"
server_key: "b24e8196caea61491c308a63b205c4d5"
server_key_iv: "04a371a516ab3e4c48ffb058909ff54a"
server_key_version: "encryptionkey@0"
I20260806 17:06:43.007347 30772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:43.012193 30772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:43.015646 30772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:43.015875 30772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:43.024475 30772 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36407
I20260806 17:06:43.024501 30842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36407 every 8 connection(s)
I20260806 17:06:43.025537 30772 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:43.028632 30843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:43.030843 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 30772
I20260806 17:06:43.030999 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:43.031309 9661 external_mini_cluster.cc:1428] Setting key 9864abbce0c04b63361aa049982feeff
I20260806 17:06:43.034942 30843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94: Bootstrap starting.
I20260806 17:06:43.036875 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:43.032670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45233 (local address 127.0.0.1:36407)
0806 17:06:43.033197 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:43.033214 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:43.033251 (+ 37us) server_negotiation.cc:407] Connection header received
0806 17:06:43.034076 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:43.034097 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:43.034394 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:43.034784 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:43.035603 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:43.035609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:43.035625 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:43.035646 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:43.035719 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:43.035740 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:43.035897 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:43.035972 (+ 75us) server_negotiation.cc:299] Negotiation successful
0806 17:06:43.036132 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":164,"threads_started":1}
I20260806 17:06:43.037657 30843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:43.038933 30843 log.cc:824] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:43.040191 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:06:43.041090 30843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94: No bootstrap required, opened a new log
I20260806 17:06:43.044230 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:43.042400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50441 (local address 127.0.0.1:36407)
0806 17:06:43.042520 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:43.042527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:43.042717 (+ 190us) server_negotiation.cc:407] Connection header received
0806 17:06:43.042878 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:43.042881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:43.042912 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:43.042991 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:43.043778 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:43.043782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:43.043785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:43.043801 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:43.043829 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:43.043839 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:43.043928 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:43.044018 (+ 90us) server_negotiation.cc:299] Negotiation successful
0806 17:06:43.044060 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 17:06:43.045748 30843 raft_consensus.cc:348] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc41bb18e654b04aa9f8e4b6c446a94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36407 } }
I20260806 17:06:43.046067 30843 raft_consensus.cc:374] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:43.046136 30843 raft_consensus.cc:729] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abc41bb18e654b04aa9f8e4b6c446a94, State: Initialized, Role: FOLLOWER
I20260806 17:06:43.046563 30843 consensus_queue.cc:260] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc41bb18e654b04aa9f8e4b6c446a94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36407 } }
I20260806 17:06:43.046720 30843 raft_consensus.cc:388] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:43.046788 30843 raft_consensus.cc:482] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:43.046890 30843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:43.047891 30843 raft_consensus.cc:504] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc41bb18e654b04aa9f8e4b6c446a94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36407 } }
I20260806 17:06:43.048357 30843 leader_election.cc:302] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abc41bb18e654b04aa9f8e4b6c446a94; no voters:
I20260806 17:06:43.048805 30843 leader_election.cc:288] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:43.049018 30853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:43.049325 30853 raft_consensus.cc:686] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94 [term 1 LEADER]: Becoming Leader. State: Replica: abc41bb18e654b04aa9f8e4b6c446a94, State: Running, Role: LEADER
I20260806 17:06:43.049642 30853 consensus_queue.cc:237] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc41bb18e654b04aa9f8e4b6c446a94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36407 } }
I20260806 17:06:43.050318 30843 sys_catalog.cc:564] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:43.051551 30855 sys_catalog.cc:455] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abc41bb18e654b04aa9f8e4b6c446a94. Latest consensus state: current_term: 1 leader_uuid: "abc41bb18e654b04aa9f8e4b6c446a94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc41bb18e654b04aa9f8e4b6c446a94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36407 } } }
I20260806 17:06:43.051548 30854 sys_catalog.cc:455] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abc41bb18e654b04aa9f8e4b6c446a94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc41bb18e654b04aa9f8e4b6c446a94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36407 } } }
I20260806 17:06:43.051699 30855 sys_catalog.cc:458] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:43.051699 30854 sys_catalog.cc:458] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:43.052161 30863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:43.055178 30863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:43.061172 30863 catalog_manager.cc:1347] Generated new cluster ID: 9c2ef08927574b59ba527101c576f54b
I20260806 17:06:43.061269 30863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:43.080466 30863 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:43.081008 30863 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:43.085386 30863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abc41bb18e654b04aa9f8e4b6c446a94: Generated new TSK 0
I20260806 17:06:43.086349 30863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:06:43.100991 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 30772
2026-08-06T17:06:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1461 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:06:43 dist-test-slave-69rg krb5kdc[30875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:06:43 dist-test-slave-69rg krb5kdc[30875](info): set up 2 sockets
Aug 06 17:06:43 dist-test-slave-69rg krb5kdc[30875](info): commencing operation
krb5kdc: starting...
W20260806 17:06:44.177557 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:06:44 dist-test-slave-69rg krb5kdc[30875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:06:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 17:06:44.312888 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:44.446430 30888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:44.446765 30888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:44.446811 30888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:44.451013 30888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:44.451086 30888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:44.451108 30888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:44.451128 30888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:44.451143 30888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:44.456723 30888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:44.457926 30888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:44.458979 30888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:44.468544 30896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:44.468670 30894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:44.469017 30893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:44.469072 30888 server_base.cc:1034] running on GCE node
I20260806 17:06:44.469732 30888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:44.471086 30888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:44.472282 30888 hybrid_clock.cc:648] HybridClock initialized: now 1786036004472267 us; error 37 us; skew 500 ppm
Aug 06 17:06:44 dist-test-slave-69rg krb5kdc[30875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:44.475265 30888 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 17:06:44.476259 30888 webserver.cc:460] Webserver started at http://127.0.0.1:41561/ using document root <none> and password file <none>
I20260806 17:06:44.476961 30888 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:44.477065 30888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:44.477319 30888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:44.479732 30888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "c7c4f0cb09e044b48974c6253abb4bf2"
format_stamp: "Formatted at 2026-08-06 17:06:44 on dist-test-slave-69rg"
server_key: "02438ce4d528a9e2e76e78ad61c826da"
server_key_iv: "237bc8f44fbaa60e2d399b4bcb06e056"
server_key_version: "encryptionkey@0"
I20260806 17:06:44.480396 30888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "c7c4f0cb09e044b48974c6253abb4bf2"
format_stamp: "Formatted at 2026-08-06 17:06:44 on dist-test-slave-69rg"
server_key: "02438ce4d528a9e2e76e78ad61c826da"
server_key_iv: "237bc8f44fbaa60e2d399b4bcb06e056"
server_key_version: "encryptionkey@0"
I20260806 17:06:44.484551 30888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 17:06:44.487265 30903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:44.488528 30888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 17:06:44.488701 30888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "c7c4f0cb09e044b48974c6253abb4bf2"
format_stamp: "Formatted at 2026-08-06 17:06:44 on dist-test-slave-69rg"
server_key: "02438ce4d528a9e2e76e78ad61c826da"
server_key_iv: "237bc8f44fbaa60e2d399b4bcb06e056"
server_key_version: "encryptionkey@0"
I20260806 17:06:44.488829 30888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:44.504871 30888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:44.508283 30888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:44.508538 30888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:44.516175 30888 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44909
I20260806 17:06:44.516203 30957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44909 every 8 connection(s)
I20260806 17:06:44.517228 30888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:44.520385 30958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:44.521026 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 30888
I20260806 17:06:44.521205 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:44.521570 9661 external_mini_cluster.cc:1428] Setting key 2869a6ceff0283c8cd4452874be20cf0
I20260806 17:06:44.526837 30958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2: Bootstrap starting.
I20260806 17:06:44.529198 30958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:44.530041 30958 log.cc:824] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2: Log is configured to *not* fsync() on all Append() calls
Aug 06 17:06:44 dist-test-slave-69rg krb5kdc[30875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 17:06:44.532420 30958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2: No bootstrap required, opened a new log
I20260806 17:06:44.536638 30961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:44.523138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46971 (local address 127.0.0.1:44909)
0806 17:06:44.523632 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:44.523648 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:44.523685 (+ 37us) server_negotiation.cc:407] Connection header received
0806 17:06:44.524426 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:44.524445 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:44.524807 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:44.525170 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:44.526043 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:44.527122 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:44.527986 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:44.528459 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:44.528621 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:44.531533 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:44.531539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:44.531552 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:44.531580 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:44.533365 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:44.533948 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:44.533953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:44.533958 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:44.534099 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:44.534501 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:44.534504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:44.534505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:44.534943 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:44.535100 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:44.535405 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 17:06:44.535852 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":155,"threads_started":1}
I20260806 17:06:44.536628 30958 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c4f0cb09e044b48974c6253abb4bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44909 } }
I20260806 17:06:44.536921 30958 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:44.536978 30958 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c4f0cb09e044b48974c6253abb4bf2, State: Initialized, Role: FOLLOWER
I20260806 17:06:44.537402 30958 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c4f0cb09e044b48974c6253abb4bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44909 } }
I20260806 17:06:44.537524 30958 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:44.537606 30958 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:44.537724 30958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:44.538982 30958 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c4f0cb09e044b48974c6253abb4bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44909 } }
I20260806 17:06:44.539319 30958 leader_election.cc:302] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7c4f0cb09e044b48974c6253abb4bf2; no voters:
I20260806 17:06:44.539651 30958 leader_election.cc:288] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:44.539739 30963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:44.539927 30963 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2 [term 1 LEADER]: Becoming Leader. State: Replica: c7c4f0cb09e044b48974c6253abb4bf2, State: Running, Role: LEADER
I20260806 17:06:44.540040 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:06:44.540218 30963 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c4f0cb09e044b48974c6253abb4bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44909 } }
I20260806 17:06:44.540962 30958 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:44.541365 30963 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7c4f0cb09e044b48974c6253abb4bf2. Latest consensus state: current_term: 1 leader_uuid: "c7c4f0cb09e044b48974c6253abb4bf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c4f0cb09e044b48974c6253abb4bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44909 } } }
I20260806 17:06:44.541472 30963 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:44.541905 30964 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7c4f0cb09e044b48974c6253abb4bf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c4f0cb09e044b48974c6253abb4bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44909 } } }
I20260806 17:06:44.542009 30964 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2 [sys.catalog]: This master's current role is: LEADER
W20260806 17:06:44.545794 30984 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:06:44.545944 30984 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:06:44.546043 30971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:44.546732 30971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:44.552819 30961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:44.542602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40187 (local address 127.0.0.1:44909)
0806 17:06:44.542733 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:44.542738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:44.543134 (+ 396us) server_negotiation.cc:407] Connection header received
0806 17:06:44.543346 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:44.543349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:44.543406 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:44.543472 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:44.544554 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:44.545407 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:44.546773 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:44.547149 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:44.547256 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:44.549310 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:44.549314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:44.549317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:44.549342 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:44.551146 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:44.551582 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:44.551588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:44.551590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:44.551675 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:44.551952 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:44.551956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:44.551958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:44.552149 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:44.552287 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:44.552519 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 17:06:44.552645 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 17:06:44.553282 30971 catalog_manager.cc:1347] Generated new cluster ID: 5e1d6243b85449f3a0039dcccca74266
I20260806 17:06:44.553368 30971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:44.563299 30971 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:44.564271 30971 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:44.568856 30971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7c4f0cb09e044b48974c6253abb4bf2: Generated new TSK 0
I20260806 17:06:44.569510 30971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:06:44.578665 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 30888
2026-08-06T17:06:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1477 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:06:44 dist-test-slave-69rg krb5kdc[30990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:06:44 dist-test-slave-69rg krb5kdc[30990](info): set up 2 sockets
Aug 06 17:06:44 dist-test-slave-69rg krb5kdc[30990](info): commencing operation
krb5kdc: starting...
W20260806 17:06:45.665472 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:06:45 dist-test-slave-69rg krb5kdc[30990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:06:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 17:06:45.798156 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:45.925612 31003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:45.926003 31003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:45.926151 31003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:45.930831 31003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:45.930954 31003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:45.930994 31003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:45.931043 31003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:45.931094 31003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:45.937037 31003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:45.938329 31003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:45.939427 31003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 17:06:45.948889 31003 server_base.cc:1034] running on GCE node
W20260806 17:06:45.948844 31008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:45.948849 31011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:45.948844 31009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:45.949774 31003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:45.951040 31003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:45.952265 31003 hybrid_clock.cc:648] HybridClock initialized: now 1786036005952229 us; error 54 us; skew 500 ppm
Aug 06 17:06:45 dist-test-slave-69rg krb5kdc[30990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:45.961704 31003 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 17:06:45.962824 31003 webserver.cc:460] Webserver started at http://127.0.0.1:45727/ using document root <none> and password file <none>
I20260806 17:06:45.963487 31003 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:45.963598 31003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:45.963855 31003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:45.966286 31003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "1764a4e6fadc4c14a25670748a02dd48"
format_stamp: "Formatted at 2026-08-06 17:06:45 on dist-test-slave-69rg"
server_key: "abfe60f58e43de6cfd9a026af04c1dfd"
server_key_iv: "d1db1a94da7dd3139b08235e4141d4af"
server_key_version: "encryptionkey@0"
I20260806 17:06:45.966900 31003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "1764a4e6fadc4c14a25670748a02dd48"
format_stamp: "Formatted at 2026-08-06 17:06:45 on dist-test-slave-69rg"
server_key: "abfe60f58e43de6cfd9a026af04c1dfd"
server_key_iv: "d1db1a94da7dd3139b08235e4141d4af"
server_key_version: "encryptionkey@0"
I20260806 17:06:45.970994 31003 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:06:45.973675 31018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:45.975097 31003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:06:45.975263 31003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "1764a4e6fadc4c14a25670748a02dd48"
format_stamp: "Formatted at 2026-08-06 17:06:45 on dist-test-slave-69rg"
server_key: "abfe60f58e43de6cfd9a026af04c1dfd"
server_key_iv: "d1db1a94da7dd3139b08235e4141d4af"
server_key_version: "encryptionkey@0"
I20260806 17:06:45.975402 31003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:45.990126 31003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:45.994328 31003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:45.994567 31003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:46.003175 31003 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34955
I20260806 17:06:46.003229 31072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34955 every 8 connection(s)
I20260806 17:06:46.004257 31003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:46.006906 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 31003
I20260806 17:06:46.007064 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:46.007395 9661 external_mini_cluster.cc:1428] Setting key 81d44adfa469f446d7b02840da6637d7
I20260806 17:06:46.007504 31073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:46.014581 31073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48: Bootstrap starting.
I20260806 17:06:46.018415 31073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:46.019640 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:46.010054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45441 (local address 127.0.0.1:34955)
0806 17:06:46.011173 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:46.011192 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:46.011232 (+ 40us) server_negotiation.cc:407] Connection header received
0806 17:06:46.011915 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:46.011934 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:46.012303 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:46.012610 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:46.013086 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:46.014206 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:46.015671 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:46.016174 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:46.016321 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:46.017477 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:46.017483 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:46.017498 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:46.018087 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:46.018167 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:46.018195 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:46.018280 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:46.018494 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 17:06:46.018771 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":944,"thread_start_us":712,"threads_started":1}
I20260806 17:06:46.019938 31073 log.cc:824] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:46.022938 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:06:46.023061 31073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48: No bootstrap required, opened a new log
I20260806 17:06:46.027570 31073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1764a4e6fadc4c14a25670748a02dd48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34955 } }
I20260806 17:06:46.028035 31073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:46.028126 31073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1764a4e6fadc4c14a25670748a02dd48, State: Initialized, Role: FOLLOWER
I20260806 17:06:46.028611 31073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1764a4e6fadc4c14a25670748a02dd48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34955 } }
I20260806 17:06:46.028803 31073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:46.028889 31073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:46.029016 31073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:46.030786 31073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1764a4e6fadc4c14a25670748a02dd48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34955 } }
I20260806 17:06:46.030992 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:46.025619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58157 (local address 127.0.0.1:34955)
0806 17:06:46.025915 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:46.025919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:46.025930 (+ 11us) server_negotiation.cc:407] Connection header received
0806 17:06:46.025963 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:46.025965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:46.026003 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:46.026090 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:46.026378 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:46.027592 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:46.028657 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:46.029016 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:46.029136 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:46.030325 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:46.030329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:46.030332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:46.030347 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:46.030383 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:46.030396 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:46.030476 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:46.030603 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 17:06:46.030758 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260806 17:06:46.031317 31073 leader_election.cc:302] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1764a4e6fadc4c14a25670748a02dd48; no voters:
I20260806 17:06:46.031773 31073 leader_election.cc:288] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:46.031883 31083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:46.032109 31083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48 [term 1 LEADER]: Becoming Leader. State: Replica: 1764a4e6fadc4c14a25670748a02dd48, State: Running, Role: LEADER
I20260806 17:06:46.032442 31083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1764a4e6fadc4c14a25670748a02dd48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34955 } }
I20260806 17:06:46.033270 31073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:46.033710 31083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1764a4e6fadc4c14a25670748a02dd48. Latest consensus state: current_term: 1 leader_uuid: "1764a4e6fadc4c14a25670748a02dd48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1764a4e6fadc4c14a25670748a02dd48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34955 } } }
I20260806 17:06:46.033865 31083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:46.034088 31083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1764a4e6fadc4c14a25670748a02dd48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1764a4e6fadc4c14a25670748a02dd48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34955 } } }
I20260806 17:06:46.034195 31083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:46.034353 31087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:46.037914 31087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:46.044518 31087 catalog_manager.cc:1347] Generated new cluster ID: ad0dcaef33594189b71a5ad93205c017
I20260806 17:06:46.044647 31087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:46.059494 31087 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:46.060935 31087 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:46.066864 31087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1764a4e6fadc4c14a25670748a02dd48: Generated new TSK 0
I20260806 17:06:46.067984 31087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:06:46.073297 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 31003
2026-08-06T17:06:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1494 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:06:46 dist-test-slave-69rg krb5kdc[31104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:06:46 dist-test-slave-69rg krb5kdc[31104](info): set up 2 sockets
Aug 06 17:06:46 dist-test-slave-69rg krb5kdc[31104](info): commencing operation
krb5kdc: starting...
W20260806 17:06:47.133361 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:06:47 dist-test-slave-69rg krb5kdc[31104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:06:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 17:06:47.273687 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42851
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:47.391491 31117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:47.391753 31117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:47.391795 31117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:47.395866 31117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:47.395941 31117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:47.395960 31117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:47.395977 31117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:47.396027 31117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:47.401520 31117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:47.402697 31117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:47.403769 31117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:47.412595 31123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:47.412595 31124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:47.412685 31126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:47.412914 31117 server_base.cc:1034] running on GCE node
I20260806 17:06:47.413558 31117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:47.414824 31117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:47.416018 31117 hybrid_clock.cc:648] HybridClock initialized: now 1786036007415997 us; error 33 us; skew 500 ppm
Aug 06 17:06:47 dist-test-slave-69rg krb5kdc[31104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:47.419106 31117 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 17:06:47.420233 31117 webserver.cc:460] Webserver started at http://127.0.0.1:44109/ using document root <none> and password file <none>
I20260806 17:06:47.420814 31117 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:47.420907 31117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:47.421149 31117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:47.423333 31117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "dc0ac36f92d54e50a88160a1a9271443"
format_stamp: "Formatted at 2026-08-06 17:06:47 on dist-test-slave-69rg"
server_key: "b24aaf607a48dbec32998f46e4c06c04"
server_key_iv: "0dc6ee98568977de77b2145820df2ec7"
server_key_version: "encryptionkey@0"
I20260806 17:06:47.423933 31117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "dc0ac36f92d54e50a88160a1a9271443"
format_stamp: "Formatted at 2026-08-06 17:06:47 on dist-test-slave-69rg"
server_key: "b24aaf607a48dbec32998f46e4c06c04"
server_key_iv: "0dc6ee98568977de77b2145820df2ec7"
server_key_version: "encryptionkey@0"
I20260806 17:06:47.427980 31117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 17:06:47.430708 31133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:47.431952 31117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:06:47.432103 31117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "dc0ac36f92d54e50a88160a1a9271443"
format_stamp: "Formatted at 2026-08-06 17:06:47 on dist-test-slave-69rg"
server_key: "b24aaf607a48dbec32998f46e4c06c04"
server_key_iv: "0dc6ee98568977de77b2145820df2ec7"
server_key_version: "encryptionkey@0"
I20260806 17:06:47.432233 31117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:47.437083 31117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:47.440346 31117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:47.440562 31117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:47.448475 31117 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42851
I20260806 17:06:47.448499 31186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42851 every 8 connection(s)
I20260806 17:06:47.449491 31117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:47.451737 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 31117
I20260806 17:06:47.451891 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:47.452203 9661 external_mini_cluster.cc:1428] Setting key 9860854a5062f1c618b3a56cceea462e
I20260806 17:06:47.452672 31187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:47.457657 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:47.453715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44261 (local address 127.0.0.1:42851)
0806 17:06:47.454225 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:47.454241 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:47.454278 (+ 37us) server_negotiation.cc:407] Connection header received
0806 17:06:47.454938 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:47.454954 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:47.455243 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:47.455591 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:47.456464 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:47.456469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:47.456480 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:47.456491 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:47.456547 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:47.456565 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:47.456727 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:47.456799 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 17:06:47.456958 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":145,"threads_started":1}
I20260806 17:06:47.459554 31187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443: Bootstrap starting.
I20260806 17:06:47.460566 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:06:47.462008 31187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:47.462896 31187 log.cc:824] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:47.464579 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:47.462744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38427 (local address 127.0.0.1:42851)
0806 17:06:47.462868 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:47.462873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:47.463220 (+ 347us) server_negotiation.cc:407] Connection header received
0806 17:06:47.463255 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:47.463257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:47.463288 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:47.463375 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:47.464108 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:47.464111 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:47.464113 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:47.464124 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:47.464199 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:47.464211 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:47.464285 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:47.464393 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 17:06:47.464428 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 17:06:47.465304 31187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443: No bootstrap required, opened a new log
I20260806 17:06:47.469063 31187 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc0ac36f92d54e50a88160a1a9271443" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42851 } }
I20260806 17:06:47.469293 31187 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:47.469331 31187 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc0ac36f92d54e50a88160a1a9271443, State: Initialized, Role: FOLLOWER
I20260806 17:06:47.469710 31187 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc0ac36f92d54e50a88160a1a9271443" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42851 } }
I20260806 17:06:47.469887 31187 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:47.469964 31187 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:47.470112 31187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:47.471108 31187 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc0ac36f92d54e50a88160a1a9271443" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42851 } }
I20260806 17:06:47.471407 31187 leader_election.cc:302] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc0ac36f92d54e50a88160a1a9271443; no voters:
I20260806 17:06:47.471676 31187 leader_election.cc:288] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:47.471881 31197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:47.472172 31197 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443 [term 1 LEADER]: Becoming Leader. State: Replica: dc0ac36f92d54e50a88160a1a9271443, State: Running, Role: LEADER
I20260806 17:06:47.472509 31197 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc0ac36f92d54e50a88160a1a9271443" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42851 } }
I20260806 17:06:47.472925 31187 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:47.473664 31197 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc0ac36f92d54e50a88160a1a9271443. Latest consensus state: current_term: 1 leader_uuid: "dc0ac36f92d54e50a88160a1a9271443" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc0ac36f92d54e50a88160a1a9271443" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42851 } } }
I20260806 17:06:47.473778 31197 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:47.474481 31198 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc0ac36f92d54e50a88160a1a9271443" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc0ac36f92d54e50a88160a1a9271443" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42851 } } }
I20260806 17:06:47.474579 31198 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:47.474848 31205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:47.478113 31205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:47.483780 31205 catalog_manager.cc:1347] Generated new cluster ID: 39a9704d8bc44b49ae3aab7008a31564
I20260806 17:06:47.483856 31205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:47.491542 31205 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:47.491963 31205 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:47.500098 31205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc0ac36f92d54e50a88160a1a9271443: Generated new TSK 0
I20260806 17:06:47.500831 31205 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:06:47.505918 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 31117
2026-08-06T17:06:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:06:47 dist-test-slave-69rg krb5kdc[31218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:06:47 dist-test-slave-69rg krb5kdc[31218](info): set up 2 sockets
Aug 06 17:06:47 dist-test-slave-69rg krb5kdc[31218](info): commencing operation
krb5kdc: starting...
W20260806 17:06:48.595252 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:06:48 dist-test-slave-69rg krb5kdc[31218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:06:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 17:06:48.733388 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44119
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:48.849319 31232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:48.849593 31232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:48.849635 31232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:48.853931 31232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:48.854013 31232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:48.854034 31232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:48.854051 31232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:48.854066 31232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:48.859642 31232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:48.860775 31232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:48.861879 31232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:48.871162 31241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:48.871266 31237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:48.871281 31238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:48.871461 31232 server_base.cc:1034] running on GCE node
I20260806 17:06:48.872225 31232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:48.873507 31232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:48.874697 31232 hybrid_clock.cc:648] HybridClock initialized: now 1786036008874661 us; error 51 us; skew 500 ppm
Aug 06 17:06:48 dist-test-slave-69rg krb5kdc[31218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:48.877861 31232 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 17:06:48.878923 31232 webserver.cc:460] Webserver started at http://127.0.0.1:35029/ using document root <none> and password file <none>
I20260806 17:06:48.879544 31232 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:48.879611 31232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:48.879838 31232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:48.882014 31232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "5727a3c3c97543b5a66d762026a85e61"
format_stamp: "Formatted at 2026-08-06 17:06:48 on dist-test-slave-69rg"
server_key: "ea5205962939ee9bbdb694e338e10b17"
server_key_iv: "f7c4ad82f5c8e8744599bd370a29f24b"
server_key_version: "encryptionkey@0"
I20260806 17:06:48.882635 31232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "5727a3c3c97543b5a66d762026a85e61"
format_stamp: "Formatted at 2026-08-06 17:06:48 on dist-test-slave-69rg"
server_key: "ea5205962939ee9bbdb694e338e10b17"
server_key_iv: "f7c4ad82f5c8e8744599bd370a29f24b"
server_key_version: "encryptionkey@0"
I20260806 17:06:48.886632 31232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:06:48.889221 31248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:48.890612 31232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:06:48.890766 31232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "5727a3c3c97543b5a66d762026a85e61"
format_stamp: "Formatted at 2026-08-06 17:06:48 on dist-test-slave-69rg"
server_key: "ea5205962939ee9bbdb694e338e10b17"
server_key_iv: "f7c4ad82f5c8e8744599bd370a29f24b"
server_key_version: "encryptionkey@0"
I20260806 17:06:48.890892 31232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:48.911262 31232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:48.914677 31232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:48.914932 31232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:48.922799 31232 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44119
I20260806 17:06:48.922827 31302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44119 every 8 connection(s)
I20260806 17:06:48.923851 31232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:48.926694 31303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:48.931273 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 31232
I20260806 17:06:48.931438 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:48.931783 9661 external_mini_cluster.cc:1428] Setting key c0782fbc0313c4b1979cbec912cb213d
I20260806 17:06:48.932778 31303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61: Bootstrap starting.
I20260806 17:06:48.935287 31303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:48.936051 31303 log.cc:824] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:48.938359 31303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61: No bootstrap required, opened a new log
Aug 06 17:06:48 dist-test-slave-69rg krb5kdc[31218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 17:06:48.942757 31303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5727a3c3c97543b5a66d762026a85e61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44119 } }
I20260806 17:06:48.943053 31303 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:48.943138 31303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5727a3c3c97543b5a66d762026a85e61, State: Initialized, Role: FOLLOWER
I20260806 17:06:48.943671 31303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5727a3c3c97543b5a66d762026a85e61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44119 } }
I20260806 17:06:48.943840 31303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:48.943938 31303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:48.944041 31303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:48.945279 31303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5727a3c3c97543b5a66d762026a85e61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44119 } }
I20260806 17:06:48.945653 31303 leader_election.cc:302] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5727a3c3c97543b5a66d762026a85e61; no voters:
I20260806 17:06:48.945995 31303 leader_election.cc:288] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:48.946120 31308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:48.946398 31308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61 [term 1 LEADER]: Becoming Leader. State: Replica: 5727a3c3c97543b5a66d762026a85e61, State: Running, Role: LEADER
I20260806 17:06:48.946719 31308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5727a3c3c97543b5a66d762026a85e61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44119 } }
I20260806 17:06:48.947405 31303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:48.947857 31306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:48.933367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:43075 (local address 127.0.0.1:44119)
0806 17:06:48.933905 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:48.933922 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:48.933960 (+ 38us) server_negotiation.cc:407] Connection header received
0806 17:06:48.934656 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:48.934677 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:48.935033 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:48.935381 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:48.936270 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:48.937499 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:48.938507 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:48.939006 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:48.939191 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:48.942322 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:48.942327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:48.942343 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:48.942372 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:48.944457 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:48.945071 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:48.945075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:48.945080 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:48.945193 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:48.945539 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:48.945541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:48.945542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:48.946096 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:48.946339 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:48.946654 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 17:06:48.946891 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":166,"threads_started":1}
I20260806 17:06:48.947949 31309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5727a3c3c97543b5a66d762026a85e61. Latest consensus state: current_term: 1 leader_uuid: "5727a3c3c97543b5a66d762026a85e61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5727a3c3c97543b5a66d762026a85e61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44119 } } }
I20260806 17:06:48.948069 31309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:48.948203 31310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5727a3c3c97543b5a66d762026a85e61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5727a3c3c97543b5a66d762026a85e61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44119 } } }
I20260806 17:06:48.948303 31310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:48.951592 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260806 17:06:48.952317 31324 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:06:48.952409 31324 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:06:48.952479 31313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:48.953274 31313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:48.959051 31313 catalog_manager.cc:1347] Generated new cluster ID: b8bf4842e050479dae20523bfb58b89a
I20260806 17:06:48.959124 31313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:48.963586 31306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:48.954473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:60007 (local address 127.0.0.1:44119)
0806 17:06:48.954581 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:48.954586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:48.954599 (+ 13us) server_negotiation.cc:407] Connection header received
0806 17:06:48.954637 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:48.954640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:48.954684 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:48.954769 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:48.955647 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:48.956395 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:48.957667 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:48.958037 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:48.958167 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:48.959998 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:48.960001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:48.960004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:48.960028 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:48.961697 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:48.962282 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:48.962285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:48.962287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:48.962368 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:48.962721 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:48.962724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:48.962726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:48.962912 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:48.963033 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:48.963337 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 17:06:48.963428 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 17:06:48.968534 31313 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:48.969599 31313 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:48.975729 31313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5727a3c3c97543b5a66d762026a85e61: Generated new TSK 0
I20260806 17:06:48.976397 31313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:06:48.980968 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 31232
W20260806 17:06:48.987154 31216 connection.cc:535] client connection to 127.0.0.1:44119 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44119): Connection reset by peer (error 104)
2026-08-06T17:06:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1474 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:06:49 dist-test-slave-69rg krb5kdc[31335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:06:49 dist-test-slave-69rg krb5kdc[31335](info): set up 2 sockets
Aug 06 17:06:49 dist-test-slave-69rg krb5kdc[31335](info): commencing operation
krb5kdc: starting...
W20260806 17:06:50.057339 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:06:50 dist-test-slave-69rg krb5kdc[31335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:06:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 17:06:50.194648 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:50.311800 31348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:50.312180 31348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:50.312280 31348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:50.316514 31348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:50.316655 31348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:50.316697 31348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:50.316735 31348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:50.316769 31348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:50.322460 31348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:50.323676 31348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:50.324764 31348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:50.333904 31354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:50.334090 31348 server_base.cc:1034] running on GCE node
W20260806 17:06:50.333911 31356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:50.333920 31353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:50.334968 31348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:50.336161 31348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:50.337357 31348 hybrid_clock.cc:648] HybridClock initialized: now 1786036010337339 us; error 33 us; skew 500 ppm
Aug 06 17:06:50 dist-test-slave-69rg krb5kdc[31335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:50.340443 31348 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 17:06:50.341419 31348 webserver.cc:460] Webserver started at http://127.0.0.1:32927/ using document root <none> and password file <none>
I20260806 17:06:50.342049 31348 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:50.342118 31348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:50.342296 31348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:50.344367 31348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "a31d7b893be745cdb230f5c12665b3e1"
format_stamp: "Formatted at 2026-08-06 17:06:50 on dist-test-slave-69rg"
server_key: "e19ab8757249a86d291efe182c84ed72"
server_key_iv: "a89f1a808034e51db6d558c6fba89694"
server_key_version: "encryptionkey@0"
I20260806 17:06:50.344919 31348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "a31d7b893be745cdb230f5c12665b3e1"
format_stamp: "Formatted at 2026-08-06 17:06:50 on dist-test-slave-69rg"
server_key: "e19ab8757249a86d291efe182c84ed72"
server_key_iv: "a89f1a808034e51db6d558c6fba89694"
server_key_version: "encryptionkey@0"
I20260806 17:06:50.348826 31348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260806 17:06:50.352424 31363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:50.353762 31348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 17:06:50.353982 31348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "a31d7b893be745cdb230f5c12665b3e1"
format_stamp: "Formatted at 2026-08-06 17:06:50 on dist-test-slave-69rg"
server_key: "e19ab8757249a86d291efe182c84ed72"
server_key_iv: "a89f1a808034e51db6d558c6fba89694"
server_key_version: "encryptionkey@0"
I20260806 17:06:50.354095 31348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:50.369905 31348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:50.373268 31348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:50.373443 31348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:50.380895 31348 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41977
I20260806 17:06:50.380896 31416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41977 every 8 connection(s)
I20260806 17:06:50.381997 31348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:50.382767 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 31348
I20260806 17:06:50.382920 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:50.383241 9661 external_mini_cluster.cc:1428] Setting key cbb0925f586382470334d43206aec758
I20260806 17:06:50.385361 31417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:50.391184 31417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1: Bootstrap starting.
I20260806 17:06:50.392593 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:50.384747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:58467 (local address 127.0.0.1:41977)
0806 17:06:50.385187 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:50.385202 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:50.385238 (+ 36us) server_negotiation.cc:407] Connection header received
0806 17:06:50.385959 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:50.385976 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:50.386322 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:50.386657 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:50.387072 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:50.388085 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:50.389018 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:50.389505 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:50.389673 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:50.390811 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:50.390816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:50.390831 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:50.391224 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:50.391318 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:50.391356 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:50.391514 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:50.391706 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 17:06:50.391865 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":137,"threads_started":1}
I20260806 17:06:50.393712 31417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:50.394704 31417 log.cc:824] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:50.396026 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:06:50.397022 31417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1: No bootstrap required, opened a new log
I20260806 17:06:50.400915 31417 raft_consensus.cc:348] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31d7b893be745cdb230f5c12665b3e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41977 } }
I20260806 17:06:50.401234 31417 raft_consensus.cc:374] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:50.401305 31417 raft_consensus.cc:729] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a31d7b893be745cdb230f5c12665b3e1, State: Initialized, Role: FOLLOWER
I20260806 17:06:50.401847 31417 consensus_queue.cc:260] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31d7b893be745cdb230f5c12665b3e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41977 } }
I20260806 17:06:50.402017 31417 raft_consensus.cc:388] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:50.402092 31417 raft_consensus.cc:482] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:50.402230 31417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:50.402588 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:50.398230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:35297 (local address 127.0.0.1:41977)
0806 17:06:50.398378 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:50.398386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:50.398556 (+ 170us) server_negotiation.cc:407] Connection header received
0806 17:06:50.398712 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:50.398721 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:50.398765 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:50.398853 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:50.399164 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:50.399693 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:50.400541 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:50.400783 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:50.400898 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:50.401925 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:50.401928 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:50.401930 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:50.401944 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:50.401970 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:50.402000 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:50.402128 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:50.402394 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 17:06:50.402437 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 17:06:50.403569 31417 raft_consensus.cc:504] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31d7b893be745cdb230f5c12665b3e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41977 } }
I20260806 17:06:50.403959 31417 leader_election.cc:302] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a31d7b893be745cdb230f5c12665b3e1; no voters:
I20260806 17:06:50.404250 31417 leader_election.cc:288] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:50.404381 31428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:50.404637 31428 raft_consensus.cc:686] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1 [term 1 LEADER]: Becoming Leader. State: Replica: a31d7b893be745cdb230f5c12665b3e1, State: Running, Role: LEADER
I20260806 17:06:50.405040 31428 consensus_queue.cc:237] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31d7b893be745cdb230f5c12665b3e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41977 } }
I20260806 17:06:50.405557 31417 sys_catalog.cc:564] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:50.406770 31430 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a31d7b893be745cdb230f5c12665b3e1. Latest consensus state: current_term: 1 leader_uuid: "a31d7b893be745cdb230f5c12665b3e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31d7b893be745cdb230f5c12665b3e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41977 } } }
I20260806 17:06:50.406906 31430 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:50.407348 31437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:50.407788 31429 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a31d7b893be745cdb230f5c12665b3e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31d7b893be745cdb230f5c12665b3e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41977 } } }
I20260806 17:06:50.407924 31429 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:50.410615 31437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:50.416620 31437 catalog_manager.cc:1347] Generated new cluster ID: ec13111ad0384fc8be98d57b7d7a099e
I20260806 17:06:50.416697 31437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:50.436909 31437 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:50.437991 31437 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:50.446026 31437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a31d7b893be745cdb230f5c12665b3e1: Generated new TSK 0
I20260806 17:06:50.446750 31437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:06:50.463080 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 31348
W20260806 17:06:50.470011 31333 connection.cc:535] client connection to 127.0.0.1:41977 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41977): Connection reset by peer (error 104)
2026-08-06T17:06:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1482 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:06:50 dist-test-slave-69rg krb5kdc[31450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:06:50 dist-test-slave-69rg krb5kdc[31450](info): set up 2 sockets
Aug 06 17:06:50 dist-test-slave-69rg krb5kdc[31450](info): commencing operation
krb5kdc: starting...
W20260806 17:06:51.561780 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:06:51 dist-test-slave-69rg krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:06:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 17:06:51.697222 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37945
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:51.810148 31464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:51.810434 31464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:51.810477 31464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:51.814561 31464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:51.814621 31464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:51.814643 31464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:51.814682 31464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:51.814713 31464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:51.820024 31464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:51.821095 31464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:51.822240 31464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:51.830859 31470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:51.830933 31464 server_base.cc:1034] running on GCE node
W20260806 17:06:51.830956 31472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:51.830861 31469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:51.831754 31464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:51.832891 31464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:51.834081 31464 hybrid_clock.cc:648] HybridClock initialized: now 1786036011834056 us; error 39 us; skew 500 ppm
Aug 06 17:06:51 dist-test-slave-69rg krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:51.841984 31464 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 17:06:51.843019 31464 webserver.cc:460] Webserver started at http://127.0.0.1:40809/ using document root <none> and password file <none>
I20260806 17:06:51.843597 31464 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:51.843693 31464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:51.843937 31464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:51.846087 31464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "e9e1247ad0684d49a1b2caf35b208194"
format_stamp: "Formatted at 2026-08-06 17:06:51 on dist-test-slave-69rg"
server_key: "2d3498fd3e68a819212eedbc3a9838fe"
server_key_iv: "ca98eac69b7ad6e018d6826e6732f003"
server_key_version: "encryptionkey@0"
I20260806 17:06:51.846717 31464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "e9e1247ad0684d49a1b2caf35b208194"
format_stamp: "Formatted at 2026-08-06 17:06:51 on dist-test-slave-69rg"
server_key: "2d3498fd3e68a819212eedbc3a9838fe"
server_key_iv: "ca98eac69b7ad6e018d6826e6732f003"
server_key_version: "encryptionkey@0"
I20260806 17:06:51.850646 31464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 17:06:51.853130 31479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:51.854527 31464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:06:51.854686 31464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "e9e1247ad0684d49a1b2caf35b208194"
format_stamp: "Formatted at 2026-08-06 17:06:51 on dist-test-slave-69rg"
server_key: "2d3498fd3e68a819212eedbc3a9838fe"
server_key_iv: "ca98eac69b7ad6e018d6826e6732f003"
server_key_version: "encryptionkey@0"
I20260806 17:06:51.854820 31464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:51.859649 31464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:51.862952 31464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:51.863226 31464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:51.871712 31464 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37945
I20260806 17:06:51.871742 31534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37945 every 8 connection(s)
I20260806 17:06:51.872795 31464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:51.874440 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 31464
I20260806 17:06:51.874593 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:51.874934 9661 external_mini_cluster.cc:1428] Setting key 071eb2d7144282330b04c79610b212d4
I20260806 17:06:51.876170 31535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:51.880393 31539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:51.876430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:49697 (local address 127.0.0.1:37945)
0806 17:06:51.876923 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:51.876938 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:51.876975 (+ 37us) server_negotiation.cc:407] Connection header received
0806 17:06:51.877608 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:51.877623 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:51.877988 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:51.878352 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:51.879221 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:51.879226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:51.879238 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:51.879249 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:51.879305 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:51.879324 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:51.879480 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:51.879549 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 17:06:51.879714 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":159,"threads_started":1}
I20260806 17:06:51.882535 31535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194: Bootstrap starting.
I20260806 17:06:51.883203 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:06:51.885011 31535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:51.886032 31535 log.cc:824] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:51.887053 31539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:51.885316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:37835 (local address 127.0.0.1:37945)
0806 17:06:51.885436 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:51.885440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:51.885628 (+ 188us) server_negotiation.cc:407] Connection header received
0806 17:06:51.885773 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:51.885775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:51.885804 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:51.885912 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:51.886630 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:51.886634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:51.886636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:51.886648 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:51.886674 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:51.886692 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:51.886783 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:51.886893 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 17:06:51.886934 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 17:06:51.888236 31535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194: No bootstrap required, opened a new log
I20260806 17:06:51.891772 31535 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e1247ad0684d49a1b2caf35b208194" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37945 } }
I20260806 17:06:51.891999 31535 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:51.892071 31535 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9e1247ad0684d49a1b2caf35b208194, State: Initialized, Role: FOLLOWER
I20260806 17:06:51.892510 31535 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e1247ad0684d49a1b2caf35b208194" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37945 } }
I20260806 17:06:51.892655 31535 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:51.892742 31535 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:51.892859 31535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:51.894022 31535 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e1247ad0684d49a1b2caf35b208194" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37945 } }
I20260806 17:06:51.894378 31535 leader_election.cc:302] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9e1247ad0684d49a1b2caf35b208194; no voters:
I20260806 17:06:51.894701 31535 leader_election.cc:288] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:51.894841 31546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:51.895079 31546 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194 [term 1 LEADER]: Becoming Leader. State: Replica: e9e1247ad0684d49a1b2caf35b208194, State: Running, Role: LEADER
I20260806 17:06:51.895421 31546 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e1247ad0684d49a1b2caf35b208194" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37945 } }
I20260806 17:06:51.895937 31535 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:51.897076 31547 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9e1247ad0684d49a1b2caf35b208194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e1247ad0684d49a1b2caf35b208194" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37945 } } }
I20260806 17:06:51.897212 31547 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:51.897593 31555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:51.897579 31548 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9e1247ad0684d49a1b2caf35b208194. Latest consensus state: current_term: 1 leader_uuid: "e9e1247ad0684d49a1b2caf35b208194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e1247ad0684d49a1b2caf35b208194" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37945 } } }
I20260806 17:06:51.897845 31548 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:51.900431 31555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:51.906282 31555 catalog_manager.cc:1347] Generated new cluster ID: 5634307b021e40ddb8153fa2196b4c12
I20260806 17:06:51.906432 31555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:51.924803 31555 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:51.925163 31555 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:51.929212 31555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9e1247ad0684d49a1b2caf35b208194: Generated new TSK 0
I20260806 17:06:51.930020 31555 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:06:51.941967 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 31464
2026-08-06T17:06:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1478 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:06:52 dist-test-slave-69rg krb5kdc[31569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:06:52 dist-test-slave-69rg krb5kdc[31569](info): set up 2 sockets
Aug 06 17:06:52 dist-test-slave-69rg krb5kdc[31569](info): commencing operation
krb5kdc: starting...
W20260806 17:06:53.037217 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:06:53 dist-test-slave-69rg krb5kdc[31569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:06:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 17:06:53.177452 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:53.293242 31582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:53.293530 31582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:53.293573 31582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:53.298162 31582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:53.298250 31582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:53.298272 31582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:53.298290 31582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:53.298305 31582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:53.305125 31582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43855
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:53.306516 31582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:53.307581 31582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:53.316802 31588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:53.316804 31590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:53.316807 31587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:53.317133 31582 server_base.cc:1034] running on GCE node
I20260806 17:06:53.317768 31582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:53.319057 31582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:53.320230 31582 hybrid_clock.cc:648] HybridClock initialized: now 1786036013320208 us; error 36 us; skew 500 ppm
Aug 06 17:06:53 dist-test-slave-69rg krb5kdc[31569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:53.323446 31582 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 17:06:53.324489 31582 webserver.cc:460] Webserver started at http://127.0.0.1:37661/ using document root <none> and password file <none>
I20260806 17:06:53.325068 31582 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:53.325140 31582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:53.325379 31582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:53.327510 31582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "a5da7632b07c456db86636dd936f7413"
format_stamp: "Formatted at 2026-08-06 17:06:53 on dist-test-slave-69rg"
server_key: "5ef173f509888d3022aca87e31910aa6"
server_key_iv: "6a3fa1bd46a64bf4ea712c99461b6e35"
server_key_version: "encryptionkey@0"
I20260806 17:06:53.328047 31582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "a5da7632b07c456db86636dd936f7413"
format_stamp: "Formatted at 2026-08-06 17:06:53 on dist-test-slave-69rg"
server_key: "5ef173f509888d3022aca87e31910aa6"
server_key_iv: "6a3fa1bd46a64bf4ea712c99461b6e35"
server_key_version: "encryptionkey@0"
I20260806 17:06:53.332176 31582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 17:06:53.335022 31598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:53.336493 31582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:06:53.336630 31582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "a5da7632b07c456db86636dd936f7413"
format_stamp: "Formatted at 2026-08-06 17:06:53 on dist-test-slave-69rg"
server_key: "5ef173f509888d3022aca87e31910aa6"
server_key_iv: "6a3fa1bd46a64bf4ea712c99461b6e35"
server_key_version: "encryptionkey@0"
I20260806 17:06:53.336740 31582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:53.360359 31582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:53.363593 31582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:53.363762 31582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:53.371870 31582 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43855
I20260806 17:06:53.371898 31653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43855 every 8 connection(s)
I20260806 17:06:53.372879 31582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:53.375491 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 31582
I20260806 17:06:53.375649 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:53.375789 31655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:53.375978 9661 external_mini_cluster.cc:1428] Setting key 74db59df23a2a71a088682541bbb208c
I20260806 17:06:53.382995 31655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413: Bootstrap starting.
I20260806 17:06:53.385932 31655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413: Neither blocks nor log segments found. Creating new log.
Aug 06 17:06:53 dist-test-slave-69rg krb5kdc[31569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 17:06:53.386731 31655 log.cc:824] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:53.388896 31655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413: No bootstrap required, opened a new log
I20260806 17:06:53.391741 31658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:53.377438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:57453 (local address 127.0.0.1:43855)
0806 17:06:53.377976 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:53.377992 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:53.378226 (+ 234us) server_negotiation.cc:407] Connection header received
0806 17:06:53.379184 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:53.379203 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:53.379556 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:53.379842 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:53.380787 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:53.381989 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:53.383225 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:53.383701 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:53.383819 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:53.386900 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:53.386907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:53.386928 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:53.386958 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:53.388709 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:53.389322 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:53.389327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:53.389332 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:53.389442 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:53.389797 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:53.389807 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:53.389809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:53.390272 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:53.390496 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:53.390794 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 17:06:53.391020 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":153,"threads_started":1}
I20260806 17:06:53.392916 31655 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5da7632b07c456db86636dd936f7413" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43855 } }
I20260806 17:06:53.393172 31655 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:53.393246 31655 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5da7632b07c456db86636dd936f7413, State: Initialized, Role: FOLLOWER
I20260806 17:06:53.393744 31655 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5da7632b07c456db86636dd936f7413" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43855 } }
I20260806 17:06:53.393975 31655 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:53.394047 31655 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:53.394165 31655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:53.394986 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:06:53.395360 31655 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5da7632b07c456db86636dd936f7413" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43855 } }
I20260806 17:06:53.395694 31655 leader_election.cc:302] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5da7632b07c456db86636dd936f7413; no voters:
I20260806 17:06:53.396010 31655 leader_election.cc:288] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:53.396142 31660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:53.396492 31660 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413 [term 1 LEADER]: Becoming Leader. State: Replica: a5da7632b07c456db86636dd936f7413, State: Running, Role: LEADER
I20260806 17:06:53.396859 31660 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5da7632b07c456db86636dd936f7413" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43855 } }
I20260806 17:06:53.397351 31655 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:53.398411 31660 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5da7632b07c456db86636dd936f7413. Latest consensus state: current_term: 1 leader_uuid: "a5da7632b07c456db86636dd936f7413" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5da7632b07c456db86636dd936f7413" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43855 } } }
I20260806 17:06:53.398603 31660 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:53.398849 31665 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5da7632b07c456db86636dd936f7413" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5da7632b07c456db86636dd936f7413" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43855 } } }
I20260806 17:06:53.398947 31665 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:53.399158 31675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:53.402420 31675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:53.407373 31658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:53.397258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:35315 (local address 127.0.0.1:43855)
0806 17:06:53.397364 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:53.397368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:53.397806 (+ 438us) server_negotiation.cc:407] Connection header received
0806 17:06:53.397966 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:53.397969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:53.398021 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:53.398128 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:53.399019 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:53.399872 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:53.401114 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:53.401365 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:53.401486 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:53.403249 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:53.403253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:53.403258 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:53.403285 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:53.405005 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:53.405594 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:53.405597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:53.405599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:53.405694 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:53.406134 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:53.406138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:53.406140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:53.406363 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:53.406514 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:53.407100 (+ 586us) server_negotiation.cc:299] Negotiation successful
0806 17:06:53.407201 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 17:06:53.408677 31675 catalog_manager.cc:1347] Generated new cluster ID: 0f9c5930d85847ffb21e7ab4ef734d7a
I20260806 17:06:53.408746 31675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:53.438076 31675 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:53.439090 31675 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:53.450486 31675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5da7632b07c456db86636dd936f7413: Generated new TSK 0
I20260806 17:06:53.451198 31675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:06:53.457129 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 31582
W20260806 17:06:53.463027 31567 connection.cc:535] client connection to 127.0.0.1:43855 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43855): Connection reset by peer (error 104)
2026-08-06T17:06:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1514 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:06:53 dist-test-slave-69rg krb5kdc[31687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:06:53 dist-test-slave-69rg krb5kdc[31687](info): set up 2 sockets
Aug 06 17:06:53 dist-test-slave-69rg krb5kdc[31687](info): commencing operation
krb5kdc: starting...
W20260806 17:06:54.579212 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:06:54 dist-test-slave-69rg krb5kdc[31687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:06:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 17:06:54.713431 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32783
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:54.832597 31701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:54.832876 31701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:54.832921 31701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:54.836993 31701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:54.837057 31701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:54.837077 31701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:54.837095 31701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:54.837162 31701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:54.842746 31701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32783
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:54.843906 31701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:54.844974 31701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:54.853880 31707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:54.853951 31706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:54.853963 31709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:54.854120 31701 server_base.cc:1034] running on GCE node
I20260806 17:06:54.854827 31701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:54.856156 31701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:54.857378 31701 hybrid_clock.cc:648] HybridClock initialized: now 1786036014857358 us; error 30 us; skew 500 ppm
Aug 06 17:06:54 dist-test-slave-69rg krb5kdc[31687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:54.860853 31701 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 17:06:54.861964 31701 webserver.cc:460] Webserver started at http://127.0.0.1:43383/ using document root <none> and password file <none>
I20260806 17:06:54.862540 31701 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:54.862617 31701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:54.862804 31701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:54.865067 31701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "3a0a745bfadd4fe4b26668294c8f460c"
format_stamp: "Formatted at 2026-08-06 17:06:54 on dist-test-slave-69rg"
server_key: "3397f5c759efc640f7f7acae08c6cb4a"
server_key_iv: "2ac4322eef1d9571cf002856f7d7df65"
server_key_version: "encryptionkey@0"
I20260806 17:06:54.865679 31701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "3a0a745bfadd4fe4b26668294c8f460c"
format_stamp: "Formatted at 2026-08-06 17:06:54 on dist-test-slave-69rg"
server_key: "3397f5c759efc640f7f7acae08c6cb4a"
server_key_iv: "2ac4322eef1d9571cf002856f7d7df65"
server_key_version: "encryptionkey@0"
I20260806 17:06:54.870102 31701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 17:06:54.872645 31717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:54.874049 31701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:06:54.874194 31701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "3a0a745bfadd4fe4b26668294c8f460c"
format_stamp: "Formatted at 2026-08-06 17:06:54 on dist-test-slave-69rg"
server_key: "3397f5c759efc640f7f7acae08c6cb4a"
server_key_iv: "2ac4322eef1d9571cf002856f7d7df65"
server_key_version: "encryptionkey@0"
I20260806 17:06:54.874310 31701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:54.887478 31701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:54.891499 31701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:54.891746 31701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:54.899883 31701 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32783
I20260806 17:06:54.899895 31770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32783 every 8 connection(s)
I20260806 17:06:54.901078 31701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:54.901607 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 31701
I20260806 17:06:54.901765 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:54.902078 9661 external_mini_cluster.cc:1428] Setting key 19bddfed73c5ec6adddd868422ece160
I20260806 17:06:54.905144 31771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:54.910903 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:54.903404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:51355 (local address 127.0.0.1:32783)
0806 17:06:54.903824 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:54.903846 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:54.903882 (+ 36us) server_negotiation.cc:407] Connection header received
0806 17:06:54.904509 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:54.904525 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:54.904861 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:54.905191 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:54.905605 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:54.906655 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:54.907553 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:54.908037 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:54.908208 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:54.909228 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:54.909233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:54.909246 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:54.909633 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:54.909689 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:54.909712 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:54.909888 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:54.910063 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 17:06:54.910202 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":134,"threads_started":1}
I20260806 17:06:54.911571 31771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c: Bootstrap starting.
I20260806 17:06:54.913892 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:06:54.914268 31771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:54.915086 31771 log.cc:824] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:54.917168 31771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c: No bootstrap required, opened a new log
I20260806 17:06:54.920486 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:54.915992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:36867 (local address 127.0.0.1:32783)
0806 17:06:54.916193 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:54.916199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:54.916327 (+ 128us) server_negotiation.cc:407] Connection header received
0806 17:06:54.916533 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:54.916536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:54.916591 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:54.916692 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:54.916976 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:54.917594 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:54.918609 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:54.918862 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:54.918988 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:54.919951 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:54.919963 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:54.919965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:54.919979 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:54.920005 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:54.920017 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:54.920138 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:54.920279 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 17:06:54.920322 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 17:06:54.921141 31771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0a745bfadd4fe4b26668294c8f460c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32783 } }
I20260806 17:06:54.921387 31771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:54.921468 31771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a0a745bfadd4fe4b26668294c8f460c, State: Initialized, Role: FOLLOWER
I20260806 17:06:54.921986 31771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0a745bfadd4fe4b26668294c8f460c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32783 } }
I20260806 17:06:54.922119 31771 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:54.922206 31771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:54.922325 31771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:54.923311 31771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0a745bfadd4fe4b26668294c8f460c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32783 } }
I20260806 17:06:54.923663 31771 leader_election.cc:302] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a0a745bfadd4fe4b26668294c8f460c; no voters:
I20260806 17:06:54.924000 31771 leader_election.cc:288] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:54.924101 31782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:54.924407 31782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c [term 1 LEADER]: Becoming Leader. State: Replica: 3a0a745bfadd4fe4b26668294c8f460c, State: Running, Role: LEADER
I20260806 17:06:54.924769 31782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0a745bfadd4fe4b26668294c8f460c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32783 } }
I20260806 17:06:54.925323 31771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:54.926503 31784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a0a745bfadd4fe4b26668294c8f460c. Latest consensus state: current_term: 1 leader_uuid: "3a0a745bfadd4fe4b26668294c8f460c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0a745bfadd4fe4b26668294c8f460c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32783 } } }
I20260806 17:06:54.926656 31784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:54.927054 31792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:54.927649 31783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a0a745bfadd4fe4b26668294c8f460c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0a745bfadd4fe4b26668294c8f460c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32783 } } }
I20260806 17:06:54.927738 31783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:54.930322 31792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:54.936123 31792 catalog_manager.cc:1347] Generated new cluster ID: c9cded1a249d4f51b9d5a7166f986391
I20260806 17:06:54.936200 31792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:54.957171 31792 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:54.958537 31792 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:54.968353 31792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a0a745bfadd4fe4b26668294c8f460c: Generated new TSK 0
I20260806 17:06:54.969185 31792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:06:54.981122 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 31701
W20260806 17:06:54.988143 31685 connection.cc:535] client connection to 127.0.0.1:32783 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32783): Connection reset by peer (error 104)
2026-08-06T17:06:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1525 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:06:55 dist-test-slave-69rg krb5kdc[31805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:06:55 dist-test-slave-69rg krb5kdc[31805](info): set up 2 sockets
Aug 06 17:06:55 dist-test-slave-69rg krb5kdc[31805](info): commencing operation
krb5kdc: starting...
W20260806 17:06:56.037442 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:06:56 dist-test-slave-69rg krb5kdc[31805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:06:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 17:06:56.193585 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46393
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:56.307806 31819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:56.308152 31819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:56.308238 31819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:56.312384 31819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:56.312505 31819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:56.312541 31819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:56.312580 31819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:56.312611 31819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:56.318187 31819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46393
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:56.319397 31819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:56.320492 31819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:56.329386 31824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:56.329389 31825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:56.329592 31819 server_base.cc:1034] running on GCE node
W20260806 17:06:56.329389 31827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:56.330394 31819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:56.331669 31819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:56.332883 31819 hybrid_clock.cc:648] HybridClock initialized: now 1786036016332852 us; error 51 us; skew 500 ppm
Aug 06 17:06:56 dist-test-slave-69rg krb5kdc[31805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:56.335951 31819 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 17:06:56.336959 31819 webserver.cc:460] Webserver started at http://127.0.0.1:34597/ using document root <none> and password file <none>
I20260806 17:06:56.337604 31819 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:56.337713 31819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:56.338019 31819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:56.340291 31819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "8f172593710840c7bc650a29fedd6905"
format_stamp: "Formatted at 2026-08-06 17:06:56 on dist-test-slave-69rg"
server_key: "89a6addcfcc4cbd097a1e89460b30960"
server_key_iv: "963b9d6d36de6744d009fe7f3d8f20b2"
server_key_version: "encryptionkey@0"
I20260806 17:06:56.340929 31819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "8f172593710840c7bc650a29fedd6905"
format_stamp: "Formatted at 2026-08-06 17:06:56 on dist-test-slave-69rg"
server_key: "89a6addcfcc4cbd097a1e89460b30960"
server_key_iv: "963b9d6d36de6744d009fe7f3d8f20b2"
server_key_version: "encryptionkey@0"
I20260806 17:06:56.344985 31819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 17:06:56.347549 31835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:56.348806 31819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 17:06:56.348978 31819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "8f172593710840c7bc650a29fedd6905"
format_stamp: "Formatted at 2026-08-06 17:06:56 on dist-test-slave-69rg"
server_key: "89a6addcfcc4cbd097a1e89460b30960"
server_key_iv: "963b9d6d36de6744d009fe7f3d8f20b2"
server_key_version: "encryptionkey@0"
I20260806 17:06:56.349100 31819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:56.353984 31819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:56.357059 31819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:56.357271 31819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:56.364756 31819 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46393
I20260806 17:06:56.364813 31891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46393 every 8 connection(s)
I20260806 17:06:56.365877 31819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:56.368803 31892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:56.371035 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 31819
I20260806 17:06:56.371309 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:56.371666 9661 external_mini_cluster.cc:1428] Setting key a38c87f6d6eee1fabd8bc2be4a99234a
I20260806 17:06:56.375663 31892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905: Bootstrap starting.
I20260806 17:06:56.377410 31895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:56.373297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:57659 (local address 127.0.0.1:46393)
0806 17:06:56.373795 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:56.373863 (+ 68us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:56.373917 (+ 54us) server_negotiation.cc:407] Connection header received
0806 17:06:56.374615 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:56.374633 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:56.374938 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:56.375236 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:56.376136 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:56.376141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:56.376154 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:56.376165 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:56.376220 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:56.376240 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:56.376364 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:56.376479 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 17:06:56.376664 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":180,"threads_started":1}
I20260806 17:06:56.378304 31892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:56.379217 31892 log.cc:824] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:56.380425 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:06:56.381389 31892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905: No bootstrap required, opened a new log
I20260806 17:06:56.384150 31895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:56.382550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:56391 (local address 127.0.0.1:46393)
0806 17:06:56.382679 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:56.382684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:56.382819 (+ 135us) server_negotiation.cc:407] Connection header received
0806 17:06:56.382930 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:56.382933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:56.382969 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:56.383051 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:56.383771 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:56.383774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:56.383777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:56.383789 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:56.383818 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:56.383829 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:56.383920 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:56.383982 (+ 62us) server_negotiation.cc:299] Negotiation successful
0806 17:06:56.384021 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 17:06:56.386129 31892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f172593710840c7bc650a29fedd6905" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260806 17:06:56.386377 31892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:56.386435 31892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f172593710840c7bc650a29fedd6905, State: Initialized, Role: FOLLOWER
I20260806 17:06:56.386873 31892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f172593710840c7bc650a29fedd6905" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260806 17:06:56.387001 31892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:56.387090 31892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:56.387223 31892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:56.388283 31892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f172593710840c7bc650a29fedd6905" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260806 17:06:56.388675 31892 leader_election.cc:302] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f172593710840c7bc650a29fedd6905; no voters:
I20260806 17:06:56.389045 31892 leader_election.cc:288] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:56.389214 31902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:56.389495 31902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905 [term 1 LEADER]: Becoming Leader. State: Replica: 8f172593710840c7bc650a29fedd6905, State: Running, Role: LEADER
I20260806 17:06:56.390334 31902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f172593710840c7bc650a29fedd6905" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260806 17:06:56.390518 31892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:56.392050 31904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f172593710840c7bc650a29fedd6905. Latest consensus state: current_term: 1 leader_uuid: "8f172593710840c7bc650a29fedd6905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f172593710840c7bc650a29fedd6905" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } } }
I20260806 17:06:56.392112 31903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f172593710840c7bc650a29fedd6905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f172593710840c7bc650a29fedd6905" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } } }
I20260806 17:06:56.392189 31904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:56.392227 31903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905 [sys.catalog]: This master's current role is: LEADER
W20260806 17:06:56.395722 31917 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:06:56.395816 31917 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:06:56.395963 31912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:56.397008 31912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:56.403265 31912 catalog_manager.cc:1347] Generated new cluster ID: 42303a27dc484b53a13447fda769a34f
I20260806 17:06:56.403362 31912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:56.421669 31912 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:56.422091 31912 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:56.426808 31912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f172593710840c7bc650a29fedd6905: Generated new TSK 0
I20260806 17:06:56.427587 31912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:06:56.440832 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 31819
2026-08-06T17:06:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1458 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:06:56 dist-test-slave-69rg krb5kdc[31923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:06:56 dist-test-slave-69rg krb5kdc[31923](info): set up 2 sockets
Aug 06 17:06:56 dist-test-slave-69rg krb5kdc[31923](info): commencing operation
krb5kdc: starting...
W20260806 17:06:57.501241 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:06:57 dist-test-slave-69rg krb5kdc[31923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:06:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:06:57.641182 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:36159
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:36159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:57.754609 31936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:57.754979 31936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:57.755067 31936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:57.759435 31936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:57.759545 31936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:57.759582 31936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:57.759635 31936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:57.759684 31936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:57.765288 31936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:36159
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:36159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:57.766548 31936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:57.767621 31936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 17:06:57.777233 31936 server_base.cc:1034] running on GCE node
W20260806 17:06:57.777170 31943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:57.777257 31942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:57.777168 31945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:57.778182 31936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:57.779425 31936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:57.780613 31936 hybrid_clock.cc:648] HybridClock initialized: now 1786036017780581 us; error 47 us; skew 500 ppm
Aug 06 17:06:57 dist-test-slave-69rg krb5kdc[31923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036017, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:57.788550 31936 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:06:57.789518 31936 webserver.cc:460] Webserver started at http://127.9.111.126:45485/ using document root <none> and password file <none>
I20260806 17:06:57.790144 31936 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:57.790216 31936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:57.790396 31936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:57.792486 31936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "6b7bd6b8338d47bd92d5eb5904b8a007"
format_stamp: "Formatted at 2026-08-06 17:06:57 on dist-test-slave-69rg"
server_key: "bbf0f84765b1e856df5dc61d90b13817"
server_key_iv: "5f489d852703555219854a764a6517e0"
server_key_version: "encryptionkey@0"
I20260806 17:06:57.793021 31936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "6b7bd6b8338d47bd92d5eb5904b8a007"
format_stamp: "Formatted at 2026-08-06 17:06:57 on dist-test-slave-69rg"
server_key: "bbf0f84765b1e856df5dc61d90b13817"
server_key_iv: "5f489d852703555219854a764a6517e0"
server_key_version: "encryptionkey@0"
I20260806 17:06:57.796957 31936 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 17:06:57.799525 31952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:57.800807 31936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 17:06:57.800961 31936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "6b7bd6b8338d47bd92d5eb5904b8a007"
format_stamp: "Formatted at 2026-08-06 17:06:57 on dist-test-slave-69rg"
server_key: "bbf0f84765b1e856df5dc61d90b13817"
server_key_iv: "5f489d852703555219854a764a6517e0"
server_key_version: "encryptionkey@0"
I20260806 17:06:57.801087 31936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:57.812572 31936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:57.815850 31936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:57.816116 31936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:57.823779 31936 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:36159
I20260806 17:06:57.823794 32005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:36159 every 8 connection(s)
I20260806 17:06:57.824791 31936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:57.827701 32006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:57.828878 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 31936
I20260806 17:06:57.829046 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:57.829353 9661 external_mini_cluster.cc:1428] Setting key 91dad26d4f9bc27cf577ec37ba9b123d
I20260806 17:06:57.834282 32006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007: Bootstrap starting.
I20260806 17:06:57.837097 32006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:57.838166 32006 log.cc:824] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007: Log is configured to *not* fsync() on all Append() calls
Aug 06 17:06:57 dist-test-slave-69rg krb5kdc[31923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:06:57.840569 32006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007: No bootstrap required, opened a new log
I20260806 17:06:57.844475 32006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7bd6b8338d47bd92d5eb5904b8a007" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36159 } }
I20260806 17:06:57.844719 32006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:57.844753 32006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7bd6b8338d47bd92d5eb5904b8a007, State: Initialized, Role: FOLLOWER
I20260806 17:06:57.845219 32009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:57.830771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54695 (local address 127.9.111.126:36159)
0806 17:06:57.831202 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:57.831218 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:57.831254 (+ 36us) server_negotiation.cc:407] Connection header received
0806 17:06:57.831882 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:57.831897 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:57.832277 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:57.832553 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:57.833570 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:57.834722 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:57.835976 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:57.836487 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:57.836620 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:57.840041 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:57.840046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:57.840061 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:57.840089 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:57.842028 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:57.842619 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:57.842625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:57.842629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:57.842752 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:57.843126 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:57.843135 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:57.843136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:57.843549 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:57.843814 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:57.844105 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 17:06:57.844353 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":147,"threads_started":1}
I20260806 17:06:57.845250 32006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7bd6b8338d47bd92d5eb5904b8a007" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36159 } }
I20260806 17:06:57.845407 32006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:57.845503 32006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:57.845634 32006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:57.846839 32006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7bd6b8338d47bd92d5eb5904b8a007" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36159 } }
I20260806 17:06:57.847208 32006 leader_election.cc:302] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b7bd6b8338d47bd92d5eb5904b8a007; no voters:
I20260806 17:06:57.847630 32006 leader_election.cc:288] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:57.847764 32011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:57.848021 32011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007 [term 1 LEADER]: Becoming Leader. State: Replica: 6b7bd6b8338d47bd92d5eb5904b8a007, State: Running, Role: LEADER
I20260806 17:06:57.848390 32011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7bd6b8338d47bd92d5eb5904b8a007" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36159 } }
I20260806 17:06:57.848635 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:06:57.848960 32006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:57.850196 32012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b7bd6b8338d47bd92d5eb5904b8a007" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7bd6b8338d47bd92d5eb5904b8a007" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36159 } } }
I20260806 17:06:57.850338 32012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:57.852645 32013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b7bd6b8338d47bd92d5eb5904b8a007. Latest consensus state: current_term: 1 leader_uuid: "6b7bd6b8338d47bd92d5eb5904b8a007" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7bd6b8338d47bd92d5eb5904b8a007" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36159 } } }
I20260806 17:06:57.852873 32013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007 [sys.catalog]: This master's current role is: LEADER
W20260806 17:06:57.854298 32031 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:06:57.854393 32031 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:06:57.854496 32024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:57.855476 32024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:57.861001 32009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:57.850974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52317 (local address 127.9.111.126:36159)
0806 17:06:57.851104 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:57.851109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:57.851386 (+ 277us) server_negotiation.cc:407] Connection header received
0806 17:06:57.851570 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:57.851576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:57.851634 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:57.851692 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:57.852781 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:57.853568 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:57.854768 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:57.855125 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:57.855245 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:57.857214 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:57.857218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:57.857222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:06:57.857258 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:57.858905 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:57.859443 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:57.859453 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:57.859456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:57.859540 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:06:57.859894 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:06:57.859898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:06:57.859899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:06:57.860202 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:57.860411 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:57.860711 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 17:06:57.860821 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 17:06:57.864514 32024 catalog_manager.cc:1347] Generated new cluster ID: 9ad0e7485bdc43c7b24f90c56eacc3d7
I20260806 17:06:57.864615 32024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:57.898193 32024 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:57.899294 32024 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:57.906540 32024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b7bd6b8338d47bd92d5eb5904b8a007: Generated new TSK 0
I20260806 17:06:57.907472 32024 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:06:57.918238 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 31936
W20260806 17:06:57.925016 31921 connection.cc:535] client connection to 127.9.111.126:36159 recv error: Network error: failed to read from TLS socket (remote: 127.9.111.126:36159): Connection reset by peer (error 104)
2026-08-06T17:06:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1478 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:06:57 dist-test-slave-69rg krb5kdc[32037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:06:57 dist-test-slave-69rg krb5kdc[32037](info): set up 2 sockets
Aug 06 17:06:57 dist-test-slave-69rg krb5kdc[32037](info): commencing operation
krb5kdc: starting...
W20260806 17:06:59.042404 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:06:59 dist-test-slave-69rg krb5kdc[32037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:06:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:06:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:06:59.186091 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:34533
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:40087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:34533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:06:59.304786 32050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:06:59.305078 32050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:06:59.305125 32050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:06:59.309468 32050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:06:59.309558 32050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:06:59.309579 32050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:06:59.309597 32050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:06:59.309612 32050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:06:59.315538 32050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:40087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:34533
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:34533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:06:59.316748 32050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:06:59.317870 32050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:06:59.326810 32058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:59.326892 32050 server_base.cc:1034] running on GCE node
W20260806 17:06:59.326781 32055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:06:59.327082 32056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:06:59.327737 32050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:06:59.328896 32050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:06:59.330099 32050 hybrid_clock.cc:648] HybridClock initialized: now 1786036019330079 us; error 35 us; skew 500 ppm
Aug 06 17:06:59 dist-test-slave-69rg krb5kdc[32037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036019, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:06:59.333086 32050 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:06:59.334318 32050 webserver.cc:460] Webserver started at http://127.9.111.126:45059/ using document root <none> and password file <none>
I20260806 17:06:59.334939 32050 fs_manager.cc:359] Metadata directory not provided
I20260806 17:06:59.335041 32050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:06:59.335295 32050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:06:59.337488 32050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "513a207c8f634e41b9991445d5dd0534"
format_stamp: "Formatted at 2026-08-06 17:06:59 on dist-test-slave-69rg"
server_key: "6aa9ce616e2319f819c23e33316ffed2"
server_key_iv: "2876e4934227323eb712b5ff482e04a9"
server_key_version: "encryptionkey@0"
I20260806 17:06:59.338135 32050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "513a207c8f634e41b9991445d5dd0534"
format_stamp: "Formatted at 2026-08-06 17:06:59 on dist-test-slave-69rg"
server_key: "6aa9ce616e2319f819c23e33316ffed2"
server_key_iv: "2876e4934227323eb712b5ff482e04a9"
server_key_version: "encryptionkey@0"
I20260806 17:06:59.342042 32050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 17:06:59.344682 32065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:06:59.346179 32050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:06:59.346356 32050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "513a207c8f634e41b9991445d5dd0534"
format_stamp: "Formatted at 2026-08-06 17:06:59 on dist-test-slave-69rg"
server_key: "6aa9ce616e2319f819c23e33316ffed2"
server_key_iv: "2876e4934227323eb712b5ff482e04a9"
server_key_version: "encryptionkey@0"
I20260806 17:06:59.346504 32050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:06:59.368566 32050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:06:59.371780 32050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:06:59.372020 32050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:06:59.379665 32050 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:34533
I20260806 17:06:59.379750 32119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:34533 every 8 connection(s)
I20260806 17:06:59.380733 32050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:06:59.383738 32120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:06:59.384563 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 32050
I20260806 17:06:59.384753 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:06:59.385069 9661 external_mini_cluster.cc:1428] Setting key 4083e44b440933d233e814191b45d4f8
I20260806 17:06:59.390386 32120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534: Bootstrap starting.
I20260806 17:06:59.393160 32120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534: Neither blocks nor log segments found. Creating new log.
I20260806 17:06:59.394100 32120 log.cc:824] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534: Log is configured to *not* fsync() on all Append() calls
I20260806 17:06:59.395177 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:59.386563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42293 (local address 127.9.111.126:34533)
0806 17:06:59.387005 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:59.387022 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:59.387080 (+ 58us) server_negotiation.cc:407] Connection header received
0806 17:06:59.387728 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:59.387744 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:59.388127 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:59.388412 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:59.388852 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:59.389938 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:59.391133 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:59.391620 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:59.391751 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:59.392805 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:59.392811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:59.392825 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:59.393374 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:59.393451 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:59.393478 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:59.393604 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:59.393880 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 17:06:59.394048 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":150,"threads_started":1}
I20260806 17:06:59.396536 32120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534: No bootstrap required, opened a new log
I20260806 17:06:59.398428 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:06:59.400676 32120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513a207c8f634e41b9991445d5dd0534" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34533 } }
I20260806 17:06:59.401005 32120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:06:59.401113 32120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513a207c8f634e41b9991445d5dd0534, State: Initialized, Role: FOLLOWER
I20260806 17:06:59.401619 32120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513a207c8f634e41b9991445d5dd0534" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34533 } }
I20260806 17:06:59.401777 32120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:06:59.401875 32120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:06:59.401979 32120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:06:59.402987 32120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513a207c8f634e41b9991445d5dd0534" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34533 } }
I20260806 17:06:59.403348 32120 leader_election.cc:302] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 513a207c8f634e41b9991445d5dd0534; no voters:
I20260806 17:06:59.403698 32120 leader_election.cc:288] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:06:59.403781 32131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:06:59.403990 32131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534 [term 1 LEADER]: Becoming Leader. State: Replica: 513a207c8f634e41b9991445d5dd0534, State: Running, Role: LEADER
I20260806 17:06:59.404340 32131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513a207c8f634e41b9991445d5dd0534" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34533 } }
I20260806 17:06:59.405093 32120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:06:59.405499 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:06:59.400814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43421 (local address 127.9.111.126:34533)
0806 17:06:59.400940 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 17:06:59.400944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:06:59.401222 (+ 278us) server_negotiation.cc:407] Connection header received
0806 17:06:59.401402 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:06:59.401405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:06:59.401455 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:06:59.401538 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:06:59.401899 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:59.402515 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:59.403443 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:06:59.403747 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:06:59.403869 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:06:59.404977 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:06:59.404980 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:06:59.404983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:06:59.404996 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:06:59.405032 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:06:59.405044 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:06:59.405144 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 17:06:59.405319 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 17:06:59.405358 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 17:06:59.406095 32133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "513a207c8f634e41b9991445d5dd0534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513a207c8f634e41b9991445d5dd0534" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34533 } } }
I20260806 17:06:59.406245 32133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:59.406566 32132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 513a207c8f634e41b9991445d5dd0534. Latest consensus state: current_term: 1 leader_uuid: "513a207c8f634e41b9991445d5dd0534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513a207c8f634e41b9991445d5dd0534" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 34533 } } }
I20260806 17:06:59.406644 32132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534 [sys.catalog]: This master's current role is: LEADER
I20260806 17:06:59.407413 32142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:06:59.410221 32142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:06:59.415650 32142 catalog_manager.cc:1347] Generated new cluster ID: 91d78ad8494140f7b3d095759308f5b5
I20260806 17:06:59.415726 32142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:06:59.426506 32142 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:06:59.427887 32142 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:06:59.434582 32142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 513a207c8f634e41b9991445d5dd0534: Generated new TSK 0
I20260806 17:06:59.435360 32142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:06:59.445397 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 32050
W20260806 17:06:59.452134 32035 connection.cc:535] client connection to 127.9.111.126:34533 recv error: Network error: failed to read from TLS socket (remote: 127.9.111.126:34533): Connection reset by peer (error 104)
2026-08-06T17:06:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1527 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:06:59 dist-test-slave-69rg krb5kdc[32156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:06:59 dist-test-slave-69rg krb5kdc[32156](info): set up 2 sockets
Aug 06 17:06:59 dist-test-slave-69rg krb5kdc[32156](info): commencing operation
krb5kdc: starting...
W20260806 17:07:00.509123 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:07:00 dist-test-slave-69rg krb5kdc[32156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:07:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:07:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:07:00.650051 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:45409
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:42377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:45409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:07:00.774984 32170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:07:00.775348 32170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:07:00.775436 32170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:07:00.780361 32170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:07:00.780478 32170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:07:00.780519 32170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:07:00.780560 32170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:07:00.780583 32170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:07:00.786208 32170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:42377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:45409
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:45409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:07:00.787314 32170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:07:00.788447 32170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:07:00.797556 32175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:07:00.797556 32178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:07:00.797556 32176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:07:00.798010 32170 server_base.cc:1034] running on GCE node
I20260806 17:07:00.798585 32170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:07:00.799777 32170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:07:00.800978 32170 hybrid_clock.cc:648] HybridClock initialized: now 1786036020800951 us; error 42 us; skew 500 ppm
Aug 06 17:07:00 dist-test-slave-69rg krb5kdc[32156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036020, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:07:00.804112 32170 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:07:00.805150 32170 webserver.cc:460] Webserver started at http://127.9.111.126:34261/ using document root <none> and password file <none>
I20260806 17:07:00.805757 32170 fs_manager.cc:359] Metadata directory not provided
I20260806 17:07:00.805899 32170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:07:00.806107 32170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:07:00.808307 32170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "1e953c0ea35a42a6a7c6d36987b7940f"
format_stamp: "Formatted at 2026-08-06 17:07:00 on dist-test-slave-69rg"
server_key: "0a782b15945397f6f608087bee475b33"
server_key_iv: "efdb313fe453c82b1908ef104473c240"
server_key_version: "encryptionkey@0"
I20260806 17:07:00.808889 32170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "1e953c0ea35a42a6a7c6d36987b7940f"
format_stamp: "Formatted at 2026-08-06 17:07:00 on dist-test-slave-69rg"
server_key: "0a782b15945397f6f608087bee475b33"
server_key_iv: "efdb313fe453c82b1908ef104473c240"
server_key_version: "encryptionkey@0"
I20260806 17:07:00.813124 32170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260806 17:07:00.816164 32186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:07:00.817765 32170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 17:07:00.818024 32170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "1e953c0ea35a42a6a7c6d36987b7940f"
format_stamp: "Formatted at 2026-08-06 17:07:00 on dist-test-slave-69rg"
server_key: "0a782b15945397f6f608087bee475b33"
server_key_iv: "efdb313fe453c82b1908ef104473c240"
server_key_version: "encryptionkey@0"
I20260806 17:07:00.818173 32170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:07:00.823285 32170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:07:00.827059 32170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:07:00.827325 32170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:07:00.836540 32170 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:45409
I20260806 17:07:00.836606 32241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:45409 every 8 connection(s)
I20260806 17:07:00.837684 32170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:07:00.838753 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 32170
I20260806 17:07:00.838922 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:07:00.839255 9661 external_mini_cluster.cc:1428] Setting key 2052013fbe79bddcdc222251c46d7119
I20260806 17:07:00.841663 32242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:07:00.846117 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:00.841025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52199 (local address 127.9.111.126:45409)
0806 17:07:00.841620 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:00.841795 (+ 175us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:00.841852 (+ 57us) server_negotiation.cc:407] Connection header received
0806 17:07:00.842589 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:00.842608 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:00.842928 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:00.843246 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:00.844562 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:00.844569 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:00.844582 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:07:00.844595 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:00.844661 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:07:00.844681 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:00.844823 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:00.845005 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 17:07:00.845225 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":293,"threads_started":1}
I20260806 17:07:00.849999 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:07:00.850673 32242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f: Bootstrap starting.
I20260806 17:07:00.853864 32242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f: Neither blocks nor log segments found. Creating new log.
I20260806 17:07:00.854715 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:00.852573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56855 (local address 127.9.111.126:45409)
0806 17:07:00.852724 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:00.852729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:00.852912 (+ 183us) server_negotiation.cc:407] Connection header received
0806 17:07:00.853010 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:00.853013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:00.853056 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:00.853164 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:00.854223 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:00.854227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:00.854230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:07:00.854243 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:00.854284 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:07:00.854299 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:00.854402 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:00.854538 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 17:07:00.854580 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 17:07:00.854965 32242 log.cc:824] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f: Log is configured to *not* fsync() on all Append() calls
I20260806 17:07:00.858572 32242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f: No bootstrap required, opened a new log
I20260806 17:07:00.865213 32242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e953c0ea35a42a6a7c6d36987b7940f" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45409 } }
I20260806 17:07:00.865727 32242 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:07:00.865870 32242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e953c0ea35a42a6a7c6d36987b7940f, State: Initialized, Role: FOLLOWER
I20260806 17:07:00.867161 32242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e953c0ea35a42a6a7c6d36987b7940f" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45409 } }
I20260806 17:07:00.867401 32242 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:07:00.867544 32242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:07:00.867707 32242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:07:00.868983 32242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e953c0ea35a42a6a7c6d36987b7940f" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45409 } }
I20260806 17:07:00.869477 32242 leader_election.cc:302] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e953c0ea35a42a6a7c6d36987b7940f; no voters:
I20260806 17:07:00.870007 32242 leader_election.cc:288] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:07:00.870134 32254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:07:00.870424 32254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f [term 1 LEADER]: Becoming Leader. State: Replica: 1e953c0ea35a42a6a7c6d36987b7940f, State: Running, Role: LEADER
I20260806 17:07:00.870802 32254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e953c0ea35a42a6a7c6d36987b7940f" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45409 } }
I20260806 17:07:00.871505 32242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:07:00.873143 32255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e953c0ea35a42a6a7c6d36987b7940f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e953c0ea35a42a6a7c6d36987b7940f" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45409 } } }
I20260806 17:07:00.873317 32255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f [sys.catalog]: This master's current role is: LEADER
I20260806 17:07:00.873251 32256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e953c0ea35a42a6a7c6d36987b7940f. Latest consensus state: current_term: 1 leader_uuid: "1e953c0ea35a42a6a7c6d36987b7940f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e953c0ea35a42a6a7c6d36987b7940f" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45409 } } }
I20260806 17:07:00.873383 32256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f [sys.catalog]: This master's current role is: LEADER
I20260806 17:07:00.874351 32263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:07:00.878533 32263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:07:00.886582 32263 catalog_manager.cc:1347] Generated new cluster ID: 2bad87ac83714585aac244c7e72c97ac
I20260806 17:07:00.886801 32263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:07:00.904978 32263 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:07:00.905541 32263 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:07:00.914700 32263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e953c0ea35a42a6a7c6d36987b7940f: Generated new TSK 0
I20260806 17:07:00.915887 32263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:07:00.921204 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 32170
2026-08-06T17:07:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1476 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:07:01 dist-test-slave-69rg krb5kdc[32275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:07:01 dist-test-slave-69rg krb5kdc[32275](info): set up 2 sockets
Aug 06 17:07:01 dist-test-slave-69rg krb5kdc[32275](info): commencing operation
krb5kdc: starting...
W20260806 17:07:02.022166 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:07:02 dist-test-slave-69rg krb5kdc[32275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:07:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:07:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:07:02.165271 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:42227
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:32977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:42227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:07:02.282167 32289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:07:02.282531 32289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:07:02.282635 32289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:07:02.286895 32289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:07:02.287005 32289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:07:02.287041 32289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:07:02.287101 32289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:07:02.287159 32289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:07:02.292949 32289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:32977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:42227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:42227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:07:02.294351 32289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:07:02.295424 32289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:07:02.304597 32295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:07:02.304550 32294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:07:02.304550 32297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:07:02.304884 32289 server_base.cc:1034] running on GCE node
I20260806 17:07:02.305567 32289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:07:02.306887 32289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:07:02.308105 32289 hybrid_clock.cc:648] HybridClock initialized: now 1786036022308083 us; error 38 us; skew 500 ppm
Aug 06 17:07:02 dist-test-slave-69rg krb5kdc[32275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036022, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:07:02.311358 32289 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:07:02.312537 32289 webserver.cc:460] Webserver started at http://127.9.111.126:44801/ using document root <none> and password file <none>
I20260806 17:07:02.313210 32289 fs_manager.cc:359] Metadata directory not provided
I20260806 17:07:02.313321 32289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:07:02.313593 32289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:07:02.316195 32289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "cab9eb94bb90479e98bfc26c1f597eee"
format_stamp: "Formatted at 2026-08-06 17:07:02 on dist-test-slave-69rg"
server_key: "0e86f3aabad40a3c51ba3d7f7057c8e7"
server_key_iv: "9f846de1cf406341abd89a2c725c5158"
server_key_version: "encryptionkey@0"
I20260806 17:07:02.316958 32289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "cab9eb94bb90479e98bfc26c1f597eee"
format_stamp: "Formatted at 2026-08-06 17:07:02 on dist-test-slave-69rg"
server_key: "0e86f3aabad40a3c51ba3d7f7057c8e7"
server_key_iv: "9f846de1cf406341abd89a2c725c5158"
server_key_version: "encryptionkey@0"
I20260806 17:07:02.321532 32289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 17:07:02.324506 32304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:07:02.326045 32289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:07:02.326224 32289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "cab9eb94bb90479e98bfc26c1f597eee"
format_stamp: "Formatted at 2026-08-06 17:07:02 on dist-test-slave-69rg"
server_key: "0e86f3aabad40a3c51ba3d7f7057c8e7"
server_key_iv: "9f846de1cf406341abd89a2c725c5158"
server_key_version: "encryptionkey@0"
I20260806 17:07:02.326376 32289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:07:02.340374 32289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:07:02.344025 32289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:07:02.344280 32289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:07:02.352437 32289 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:42227
I20260806 17:07:02.352466 32358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:42227 every 8 connection(s)
I20260806 17:07:02.353586 32289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:07:02.356685 32359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:07:02.362356 32359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee: Bootstrap starting.
I20260806 17:07:02.363545 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 32289
I20260806 17:07:02.363714 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:07:02.364022 9661 external_mini_cluster.cc:1428] Setting key 24acd98090fe20167b9017555a7de2cd
I20260806 17:07:02.365136 32359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee: Neither blocks nor log segments found. Creating new log.
I20260806 17:07:02.366094 32359 log.cc:824] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee: Log is configured to *not* fsync() on all Append() calls
I20260806 17:07:02.368036 32359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee: No bootstrap required, opened a new log
I20260806 17:07:02.372424 32359 raft_consensus.cc:348] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab9eb94bb90479e98bfc26c1f597eee" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42227 } }
I20260806 17:07:02.372681 32359 raft_consensus.cc:374] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:07:02.372764 32359 raft_consensus.cc:729] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab9eb94bb90479e98bfc26c1f597eee, State: Initialized, Role: FOLLOWER
I20260806 17:07:02.373262 32359 consensus_queue.cc:260] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab9eb94bb90479e98bfc26c1f597eee" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42227 } }
I20260806 17:07:02.373433 32359 raft_consensus.cc:388] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:07:02.373524 32359 raft_consensus.cc:482] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:07:02.373625 32359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee [term 0 FOLLOWER]: Advancing to term 1
Aug 06 17:07:02 dist-test-slave-69rg krb5kdc[32275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:07:02.374769 32359 raft_consensus.cc:504] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab9eb94bb90479e98bfc26c1f597eee" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42227 } }
I20260806 17:07:02.375159 32359 leader_election.cc:302] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cab9eb94bb90479e98bfc26c1f597eee; no voters:
I20260806 17:07:02.375638 32359 leader_election.cc:288] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:07:02.375815 32364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:07:02.376132 32364 raft_consensus.cc:686] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee [term 1 LEADER]: Becoming Leader. State: Replica: cab9eb94bb90479e98bfc26c1f597eee, State: Running, Role: LEADER
I20260806 17:07:02.376545 32364 consensus_queue.cc:237] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab9eb94bb90479e98bfc26c1f597eee" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42227 } }
I20260806 17:07:02.377136 32359 sys_catalog.cc:564] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:07:02.377898 32366 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee [sys.catalog]: SysCatalogTable state changed. Reason: New leader cab9eb94bb90479e98bfc26c1f597eee. Latest consensus state: current_term: 1 leader_uuid: "cab9eb94bb90479e98bfc26c1f597eee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab9eb94bb90479e98bfc26c1f597eee" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42227 } } }
I20260806 17:07:02.378759 32366 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee [sys.catalog]: This master's current role is: LEADER
I20260806 17:07:02.378619 32365 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cab9eb94bb90479e98bfc26c1f597eee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab9eb94bb90479e98bfc26c1f597eee" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 42227 } } }
I20260806 17:07:02.379119 32365 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee [sys.catalog]: This master's current role is: LEADER
I20260806 17:07:02.379439 32374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:07:02.381520 32362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:02.365248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44137 (local address 127.9.111.126:42227)
0806 17:07:02.365760 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:02.365777 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:02.365861 (+ 84us) server_negotiation.cc:407] Connection header received
0806 17:07:02.366629 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:02.366649 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:02.367003 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:02.367368 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:02.368304 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:02.369807 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:02.370791 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:02.371586 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:02.371772 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:02.374855 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:02.374864 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:02.374883 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:02.374917 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:02.377127 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:02.377794 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:02.377801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:02.377807 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:02.378003 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:02.378420 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:02.378424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:02.378426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:02.379003 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:02.379212 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:02.380278 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0806 17:07:02.380569 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":176,"threads_started":1}
I20260806 17:07:02.382908 32374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:07:02.385126 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:07:02.389457 32374 catalog_manager.cc:1347] Generated new cluster ID: 5557a1db1c42491fa43988cbde3d7903
I20260806 17:07:02.389559 32374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:07:02.399513 32362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:02.388007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57091 (local address 127.9.111.126:42227)
0806 17:07:02.388138 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:02.388143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:02.389964 (+ 1821us) server_negotiation.cc:407] Connection header received
0806 17:07:02.390102 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:02.390106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:02.390167 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:02.390278 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:02.391192 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:02.392094 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:02.393307 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:02.393733 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:02.393906 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:02.395589 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:02.395594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:02.395597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:02.395635 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:02.397530 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:02.398092 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:02.398116 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:02.398119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:02.398227 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:02.398528 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:02.398532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:02.398534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:02.398782 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:02.398925 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:02.399167 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 17:07:02.399320 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 17:07:02.411408 32374 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:07:02.412663 32374 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:07:02.419952 32374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cab9eb94bb90479e98bfc26c1f597eee: Generated new TSK 0
I20260806 17:07:02.420933 32374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:07:02.428803 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 32289
2026-08-06T17:07:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1506 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:07:02 dist-test-slave-69rg krb5kdc[32390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:07:02 dist-test-slave-69rg krb5kdc[32390](info): set up 2 sockets
Aug 06 17:07:02 dist-test-slave-69rg krb5kdc[32390](info): commencing operation
krb5kdc: starting...
W20260806 17:07:03.501310 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:07:03 dist-test-slave-69rg krb5kdc[32390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:07:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:07:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:07:03.635249 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:45443
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:38345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:45443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:07:03.749557 32403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:07:03.749935 32403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:07:03.750053 32403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:07:03.754117 32403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:07:03.754237 32403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:07:03.754277 32403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:07:03.754313 32403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:07:03.754345 32403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:07:03.759909 32403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:38345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:45443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:45443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:07:03.761067 32403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:07:03.762146 32403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:07:03.770984 32408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:07:03.771001 32409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:07:03.771044 32403 server_base.cc:1034] running on GCE node
W20260806 17:07:03.770991 32411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:07:03.771941 32403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:07:03.773168 32403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:07:03.774401 32403 hybrid_clock.cc:648] HybridClock initialized: now 1786036023774367 us; error 49 us; skew 500 ppm
Aug 06 17:07:03 dist-test-slave-69rg krb5kdc[32390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036023, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:07:03.781221 32403 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:07:03.782407 32403 webserver.cc:460] Webserver started at http://127.9.111.126:37613/ using document root <none> and password file <none>
I20260806 17:07:03.783033 32403 fs_manager.cc:359] Metadata directory not provided
I20260806 17:07:03.783133 32403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:07:03.783390 32403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:07:03.785599 32403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "d9f65274fa744489900413f361dd3e2a"
format_stamp: "Formatted at 2026-08-06 17:07:03 on dist-test-slave-69rg"
server_key: "c5a5d5b4af506324581efa3ae3f4fd03"
server_key_iv: "71175f6bd1ea90a1f8874addf4b1f52b"
server_key_version: "encryptionkey@0"
I20260806 17:07:03.786268 32403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "d9f65274fa744489900413f361dd3e2a"
format_stamp: "Formatted at 2026-08-06 17:07:03 on dist-test-slave-69rg"
server_key: "c5a5d5b4af506324581efa3ae3f4fd03"
server_key_iv: "71175f6bd1ea90a1f8874addf4b1f52b"
server_key_version: "encryptionkey@0"
I20260806 17:07:03.790244 32403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:07:03.792786 32419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:07:03.794080 32403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:07:03.794263 32403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "d9f65274fa744489900413f361dd3e2a"
format_stamp: "Formatted at 2026-08-06 17:07:03 on dist-test-slave-69rg"
server_key: "c5a5d5b4af506324581efa3ae3f4fd03"
server_key_iv: "71175f6bd1ea90a1f8874addf4b1f52b"
server_key_version: "encryptionkey@0"
I20260806 17:07:03.794390 32403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:07:03.813617 32403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:07:03.817050 32403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:07:03.817279 32403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:07:03.824745 32403 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:45443
I20260806 17:07:03.824774 32471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:45443 every 8 connection(s)
I20260806 17:07:03.825681 32403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:07:03.828437 32472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:07:03.832901 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 32403
I20260806 17:07:03.833055 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:07:03.833340 9661 external_mini_cluster.cc:1428] Setting key ef8fff9e857a490e7234d010c9ded729
I20260806 17:07:03.834444 32472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a: Bootstrap starting.
I20260806 17:07:03.836946 32472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a: Neither blocks nor log segments found. Creating new log.
I20260806 17:07:03.837742 32472 log.cc:824] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a: Log is configured to *not* fsync() on all Append() calls
I20260806 17:07:03.839713 32472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a: No bootstrap required, opened a new log
I20260806 17:07:03.842689 32475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:03.834990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45789 (local address 127.9.111.126:45443)
0806 17:07:03.835441 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:03.835457 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:03.835492 (+ 35us) server_negotiation.cc:407] Connection header received
0806 17:07:03.836164 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:03.836181 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:03.836541 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:03.836888 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:03.837320 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:03.838444 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:03.839363 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:03.839873 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:03.840054 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:03.840943 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:03.840953 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:03.840966 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:07:03.841354 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:03.841410 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:07:03.841434 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:03.841548 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:03.841666 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 17:07:03.841910 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":147,"threads_started":1}
I20260806 17:07:03.843665 32472 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f65274fa744489900413f361dd3e2a" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45443 } }
I20260806 17:07:03.843947 32472 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:07:03.844029 32472 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f65274fa744489900413f361dd3e2a, State: Initialized, Role: FOLLOWER
I20260806 17:07:03.844489 32472 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f65274fa744489900413f361dd3e2a" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45443 } }
I20260806 17:07:03.844676 32472 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:07:03.844771 32472 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:07:03.844870 32472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:07:03.845672 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:07:03.846048 32472 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f65274fa744489900413f361dd3e2a" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45443 } }
I20260806 17:07:03.846390 32472 leader_election.cc:302] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9f65274fa744489900413f361dd3e2a; no voters:
I20260806 17:07:03.846688 32472 leader_election.cc:288] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:07:03.846813 32477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:07:03.847059 32477 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a [term 1 LEADER]: Becoming Leader. State: Replica: d9f65274fa744489900413f361dd3e2a, State: Running, Role: LEADER
I20260806 17:07:03.847494 32477 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f65274fa744489900413f361dd3e2a" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45443 } }
I20260806 17:07:03.848017 32472 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:07:03.849154 32478 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9f65274fa744489900413f361dd3e2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f65274fa744489900413f361dd3e2a" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45443 } } }
I20260806 17:07:03.849303 32478 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a [sys.catalog]: This master's current role is: LEADER
I20260806 17:07:03.849710 32480 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9f65274fa744489900413f361dd3e2a. Latest consensus state: current_term: 1 leader_uuid: "d9f65274fa744489900413f361dd3e2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f65274fa744489900413f361dd3e2a" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 45443 } } }
I20260806 17:07:03.849889 32480 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a [sys.catalog]: This master's current role is: LEADER
I20260806 17:07:03.853091 32475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:03.848141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33793 (local address 127.9.111.126:45443)
0806 17:07:03.848268 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:03.848272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:03.848560 (+ 288us) server_negotiation.cc:407] Connection header received
0806 17:07:03.848763 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:03.848767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:03.848824 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:03.848884 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:03.849352 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:03.850179 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:03.851401 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:03.851636 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:03.851740 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:03.852596 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:03.852600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:03.852602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:07:03.852616 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:03.852650 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:07:03.852661 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:03.852742 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:03.852870 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 17:07:03.852924 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260806 17:07:03.853526 32498 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:07:03.853607 32498 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:07:03.853724 32492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:07:03.854643 32492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:07:03.860560 32492 catalog_manager.cc:1347] Generated new cluster ID: e5e390adeb564620b302333feba2c78b
I20260806 17:07:03.860644 32492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:07:03.882282 32492 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:07:03.883313 32492 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:07:03.892170 32492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9f65274fa744489900413f361dd3e2a: Generated new TSK 0
I20260806 17:07:03.893023 32492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:07:03.906800 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 32403
2026-08-06T17:07:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1477 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:07:03 dist-test-slave-69rg krb5kdc[32505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:07:03 dist-test-slave-69rg krb5kdc[32505](info): set up 2 sockets
Aug 06 17:07:03 dist-test-slave-69rg krb5kdc[32505](info): commencing operation
krb5kdc: starting...
W20260806 17:07:04.977604 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:07:05 dist-test-slave-69rg krb5kdc[32505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:07:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:07:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:07:05.111099 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:46749
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:46749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:07:05.227324 32519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:07:05.227676 32519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:07:05.227802 32519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:07:05.232201 32519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:07:05.232311 32519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:07:05.232350 32519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:07:05.232414 32519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:07:05.232453 32519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:07:05.238133 32519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:46749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:46749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:07:05.239269 32519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:07:05.240322 32519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:07:05.249375 32525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:07:05.249539 32519 server_base.cc:1034] running on GCE node
W20260806 17:07:05.249399 32524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:07:05.249788 32527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:07:05.250483 32519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:07:05.251694 32519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:07:05.252923 32519 hybrid_clock.cc:648] HybridClock initialized: now 1786036025252893 us; error 44 us; skew 500 ppm
Aug 06 17:07:05 dist-test-slave-69rg krb5kdc[32505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036025, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:07:05.256357 32519 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:07:05.257478 32519 webserver.cc:460] Webserver started at http://127.9.111.126:42027/ using document root <none> and password file <none>
I20260806 17:07:05.258175 32519 fs_manager.cc:359] Metadata directory not provided
I20260806 17:07:05.258296 32519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:07:05.258553 32519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:07:05.260751 32519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "128cfe7b587e4520945749f8bd331ad2"
format_stamp: "Formatted at 2026-08-06 17:07:05 on dist-test-slave-69rg"
server_key: "740a09a1527c110f46c1c637740c74d8"
server_key_iv: "4461e62c237c6e90a5f03af65797f2f5"
server_key_version: "encryptionkey@0"
I20260806 17:07:05.261371 32519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "128cfe7b587e4520945749f8bd331ad2"
format_stamp: "Formatted at 2026-08-06 17:07:05 on dist-test-slave-69rg"
server_key: "740a09a1527c110f46c1c637740c74d8"
server_key_iv: "4461e62c237c6e90a5f03af65797f2f5"
server_key_version: "encryptionkey@0"
I20260806 17:07:05.265708 32519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 17:07:05.268779 32535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:07:05.270467 32519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:07:05.270658 32519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "128cfe7b587e4520945749f8bd331ad2"
format_stamp: "Formatted at 2026-08-06 17:07:05 on dist-test-slave-69rg"
server_key: "740a09a1527c110f46c1c637740c74d8"
server_key_iv: "4461e62c237c6e90a5f03af65797f2f5"
server_key_version: "encryptionkey@0"
I20260806 17:07:05.270818 32519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:07:05.276360 32519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:07:05.280450 32519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:07:05.280799 32519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:07:05.289136 32519 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:46749
I20260806 17:07:05.289166 32590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:46749 every 8 connection(s)
I20260806 17:07:05.290309 32519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:07:05.294191 32591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:07:05.299064 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 32519
I20260806 17:07:05.299209 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:07:05.299537 9661 external_mini_cluster.cc:1428] Setting key 5e20238b78563b256cebec1d5e265ef2
I20260806 17:07:05.301651 32591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2: Bootstrap starting.
I20260806 17:07:05.304248 32591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2: Neither blocks nor log segments found. Creating new log.
I20260806 17:07:05.305303 32591 log.cc:824] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2: Log is configured to *not* fsync() on all Append() calls
I20260806 17:07:05.305684 32594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:05.301169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49699 (local address 127.9.111.126:46749)
0806 17:07:05.301666 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:05.301682 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:05.301720 (+ 38us) server_negotiation.cc:407] Connection header received
0806 17:07:05.302533 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:05.302553 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:05.302865 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:05.303245 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:05.304117 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:05.304122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:05.304135 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:07:05.304146 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:05.304202 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:07:05.304259 (+ 57us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:05.304388 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:05.304503 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 17:07:05.304670 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":141,"threads_started":1}
I20260806 17:07:05.307780 32591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2: No bootstrap required, opened a new log
I20260806 17:07:05.308823 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:07:05.312063 32591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128cfe7b587e4520945749f8bd331ad2" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 46749 } }
I20260806 17:07:05.312443 32591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 17:07:05.312536 32591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128cfe7b587e4520945749f8bd331ad2, State: Initialized, Role: FOLLOWER
I20260806 17:07:05.312937 32594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:05.311120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43235 (local address 127.9.111.126:46749)
0806 17:07:05.311256 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:05.311261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:05.311390 (+ 129us) server_negotiation.cc:407] Connection header received
0806 17:07:05.311527 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:05.311529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:05.311563 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:05.311623 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:05.312444 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:05.312448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:05.312454 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 17:07:05.312471 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:05.312510 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 17:07:05.312539 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:05.312639 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:05.312737 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 17:07:05.312792 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 17:07:05.313073 32591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128cfe7b587e4520945749f8bd331ad2" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 46749 } }
I20260806 17:07:05.313261 32591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 17:07:05.313345 32591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 17:07:05.313449 32591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:07:05.314661 32591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128cfe7b587e4520945749f8bd331ad2" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 46749 } }
I20260806 17:07:05.315032 32591 leader_election.cc:302] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 128cfe7b587e4520945749f8bd331ad2; no voters:
I20260806 17:07:05.315400 32591 leader_election.cc:288] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 17:07:05.315539 32602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:07:05.315822 32602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2 [term 1 LEADER]: Becoming Leader. State: Replica: 128cfe7b587e4520945749f8bd331ad2, State: Running, Role: LEADER
I20260806 17:07:05.316143 32602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128cfe7b587e4520945749f8bd331ad2" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 46749 } }
I20260806 17:07:05.316828 32591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:07:05.318524 32604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 128cfe7b587e4520945749f8bd331ad2. Latest consensus state: current_term: 1 leader_uuid: "128cfe7b587e4520945749f8bd331ad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128cfe7b587e4520945749f8bd331ad2" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 46749 } } }
I20260806 17:07:05.318585 32603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "128cfe7b587e4520945749f8bd331ad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128cfe7b587e4520945749f8bd331ad2" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 46749 } } }
I20260806 17:07:05.318728 32603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2 [sys.catalog]: This master's current role is: LEADER
I20260806 17:07:05.318663 32604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2 [sys.catalog]: This master's current role is: LEADER
W20260806 17:07:05.322268 32617 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:07:05.322484 32617 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:07:05.322681 32615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:07:05.323678 32615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:07:05.330351 32615 catalog_manager.cc:1347] Generated new cluster ID: cbb943391a41406d87a72bbc5dfa9935
I20260806 17:07:05.330506 32615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:07:05.352002 32615 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:07:05.352386 32615 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:07:05.358489 32615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 128cfe7b587e4520945749f8bd331ad2: Generated new TSK 0
I20260806 17:07:05.359318 32615 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 17:07:05.374835 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 32519
2026-08-06T17:07:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1467 ms)
[----------] 18 tests from AuthTokenIssuingTest (26687 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:07:05 dist-test-slave-69rg krb5kdc[32623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:07:05 dist-test-slave-69rg krb5kdc[32623](info): set up 2 sockets
Aug 06 17:07:05 dist-test-slave-69rg krb5kdc[32623](info): commencing operation
krb5kdc: starting...
W20260806 17:07:06.465641 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:07:06 dist-test-slave-69rg krb5kdc[32623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:07:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:07:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:07:06.607936 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:36141
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:43645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:36141,127.9.111.125:36099,127.9.111.124:41739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:07:06.731211 32637 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 17:07:06.731595 32637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:07:06.731669 32637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:07:06.731745 32637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:07:06.736207 32637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:07:06.736333 32637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:07:06.736371 32637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:07:06.736414 32637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:07:06.736451 32637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:07:06.742372 32637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:43645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:36141,127.9.111.125:36099,127.9.111.124:41739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:36141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:07:06.743614 32637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:07:06.744678 32637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 17:07:06.753973 32637 server_base.cc:1034] running on GCE node
W20260806 17:07:06.753890 32643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:07:06.753890 32645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:07:06.753886 32642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:07:06.754918 32637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:07:06.756209 32637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:07:06.757550 32637 hybrid_clock.cc:648] HybridClock initialized: now 1786036026757517 us; error 48 us; skew 500 ppm
Aug 06 17:07:06 dist-test-slave-69rg krb5kdc[32623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036026, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:07:06.761015 32637 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:07:06.762189 32637 webserver.cc:460] Webserver started at http://127.9.111.126:40913/ using document root <none> and password file <none>
I20260806 17:07:06.762820 32637 fs_manager.cc:359] Metadata directory not provided
I20260806 17:07:06.762890 32637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:07:06.763134 32637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:07:06.765352 32637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "37846176c33848e1b5cb06ae8fc76466"
format_stamp: "Formatted at 2026-08-06 17:07:06 on dist-test-slave-69rg"
server_key: "f66c3774235a63270f017d094885a1dd"
server_key_iv: "6ff68c2ddd79f6c0a1aca3bb997a3f85"
server_key_version: "encryptionkey@0"
I20260806 17:07:06.766017 32637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "37846176c33848e1b5cb06ae8fc76466"
format_stamp: "Formatted at 2026-08-06 17:07:06 on dist-test-slave-69rg"
server_key: "f66c3774235a63270f017d094885a1dd"
server_key_iv: "6ff68c2ddd79f6c0a1aca3bb997a3f85"
server_key_version: "encryptionkey@0"
I20260806 17:07:06.770157 32637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:07:06.772876 32652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:07:06.774407 32637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:07:06.774581 32637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "37846176c33848e1b5cb06ae8fc76466"
format_stamp: "Formatted at 2026-08-06 17:07:06 on dist-test-slave-69rg"
server_key: "f66c3774235a63270f017d094885a1dd"
server_key_iv: "6ff68c2ddd79f6c0a1aca3bb997a3f85"
server_key_version: "encryptionkey@0"
I20260806 17:07:06.774716 32637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:07:06.785908 32637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:07:06.789534 32637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:07:06.789861 32637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:07:06.797917 32637 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:36141
I20260806 17:07:06.798241 32705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:36141 every 8 connection(s)
I20260806 17:07:06.799000 32637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:07:06.801093 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 32637
I20260806 17:07:06.801283 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:07:06.801589 9661 external_mini_cluster.cc:1428] Setting key dc461d5e0970490d252b572362af8bf7
I20260806 17:07:06.802165 32706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:07:06.806219 32706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.111.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.125@KRBTEST.COM" created.
Aug 06 17:07:06 dist-test-slave-69rg krb5kdc[32623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036026, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:07:06.823665 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:06.808574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:36141 (local address 127.0.0.1:41628)
0806 17:07:06.809079 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:06.809132 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 17:07:06.810079 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:07:06.810902 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:07:06.810906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:07:06.811259 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:07:06.811639 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:06.811663 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:06.812940 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:06.812943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:06.813910 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:06.813920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:06.814826 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:06.814829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:06.815042 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:06.815804 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:07:06.815821 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:07:06.817488 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:07:06.820190 (+ 2702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:06.820200 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:06.820209 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:06.820470 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:06.820939 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:06.820942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:06.820943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:06.821085 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:06.822073 (+ 988us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:07:06.822079 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:07:06.822264 (+ 185us) client_negotiation.cc:769] Sending connection context
0806 17:07:06.822442 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 17:07:06.822763 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":59,"thread_start_us":154,"threads_started":1}
I20260806 17:07:06.823669 32710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:06.808574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41628 (local address 127.9.111.126:36141)
0806 17:07:06.809160 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:06.809175 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:06.809221 (+ 46us) server_negotiation.cc:407] Connection header received
0806 17:07:06.810434 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:06.810442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:06.810691 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:06.810863 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:06.811809 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:06.812802 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:06.814035 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:06.814639 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:06.814905 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:06.817668 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:06.817674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:06.817689 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:06.817717 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:06.820040 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:06.820599 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:06.820610 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:06.820613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:06.820799 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:06.821262 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:06.821267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:06.821269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:06.821938 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:06.822096 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:06.822558 (+ 462us) server_negotiation.cc:299] Negotiation successful
0806 17:07:06.822871 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":436,"spinlock_wait_cycles":3968,"thread_start_us":145,"threads_started":1}
I20260806 17:07:06.826401 32706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } has no permanent_uuid. Determining permanent_uuid...
I20260806 17:07:06.827242 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:06.826924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:36099 (local address 127.0.0.1:39284)
0806 17:07:06.827054 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:06.827132 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 17:07:06.827453 32655 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.111.125:36099: connect: Connection refused (error 111)
I20260806 17:07:06.828512 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:06.828285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:36099 (local address 127.0.0.1:39298)
0806 17:07:06.828399 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:06.828452 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 17:07:06.828758 32706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.125:36099: Network error: Client connection negotiation failed: client connection to 127.9.111.125:36099: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.125.keytab.
Entry for principal kudu/127.9.111.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.125.keytab.
I20260806 17:07:06.834442 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.125:36099
--webserver_interface=127.9.111.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.125.keytab
--principal=kudu/127.9.111.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:43645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:36141,127.9.111.125:36099,127.9.111.124:41739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:07:06.878182 32706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } attempt: 1
I20260806 17:07:06.879386 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:06.878960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:36099 (local address 127.0.0.1:39306)
0806 17:07:06.879229 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:06.879298 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260806 17:07:06.880290 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:06.880033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:36099 (local address 127.0.0.1:39322)
0806 17:07:06.880173 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:06.880233 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260806 17:07:06.880519 32706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.125:36099: Network error: Client connection negotiation failed: client connection to 127.9.111.125:36099: connect: Connection refused (error 111)
I20260806 17:07:06.948710 32706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } attempt: 2
I20260806 17:07:06.949600 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:06.949291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:36099 (local address 127.0.0.1:39334)
0806 17:07:06.949447 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:06.949520 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 17:07:06.950536 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:06.950200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:36099 (local address 127.0.0.1:39344)
0806 17:07:06.950363 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:06.950442 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 17:07:06.950729 32706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.125:36099: Network error: Client connection negotiation failed: client connection to 127.9.111.125:36099: connect: Connection refused (error 111)
W20260806 17:07:06.953248 32713 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 17:07:06.953523 32713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:07:06.953600 32713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:07:06.953701 32713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:07:06.957881 32713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:07:06.957944 32713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:07:06.957964 32713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:07:06.958014 32713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:07:06.958046 32713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:07:06.963557 32713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:43645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:36141,127.9.111.125:36099,127.9.111.124:41739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.125.keytab
--principal=kudu/127.9.111.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.125:36099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.111.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:07:06.964699 32713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:07:06.965742 32713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:07:06.974606 32718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:07:06.974599 32719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:07:06.974602 32721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:07:06.975090 32713 server_base.cc:1034] running on GCE node
I20260806 17:07:06.975672 32713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:07:06.976805 32713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:07:06.977982 32713 hybrid_clock.cc:648] HybridClock initialized: now 1786036026977963 us; error 35 us; skew 500 ppm
Aug 06 17:07:06 dist-test-slave-69rg krb5kdc[32623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036026, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:07:06.981170 32713 init.cc:377] Logged in from keytab as kudu/127.9.111.125@KRBTEST.COM (short username kudu)
I20260806 17:07:06.982276 32713 webserver.cc:460] Webserver started at http://127.9.111.125:33799/ using document root <none> and password file <none>
I20260806 17:07:06.982895 32713 fs_manager.cc:359] Metadata directory not provided
I20260806 17:07:06.982993 32713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:07:06.983239 32713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:07:06.985399 32713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-1/data/instance:
uuid: "8cfb28f0db9f41e7a6728a07514be3c7"
format_stamp: "Formatted at 2026-08-06 17:07:06 on dist-test-slave-69rg"
server_key: "48fe0b674a3edfaa1dbb8c070163f226"
server_key_iv: "72f9af3973ec4a64c8843d25320a3647"
server_key_version: "encryptionkey@0"
I20260806 17:07:06.986057 32713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-1/wal/instance:
uuid: "8cfb28f0db9f41e7a6728a07514be3c7"
format_stamp: "Formatted at 2026-08-06 17:07:06 on dist-test-slave-69rg"
server_key: "48fe0b674a3edfaa1dbb8c070163f226"
server_key_iv: "72f9af3973ec4a64c8843d25320a3647"
server_key_version: "encryptionkey@0"
I20260806 17:07:06.990003 32713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 17:07:06.992563 32728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:07:06.993883 32713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 17:07:06.994033 32713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-1/wal
uuid: "8cfb28f0db9f41e7a6728a07514be3c7"
format_stamp: "Formatted at 2026-08-06 17:07:06 on dist-test-slave-69rg"
server_key: "48fe0b674a3edfaa1dbb8c070163f226"
server_key_iv: "72f9af3973ec4a64c8843d25320a3647"
server_key_version: "encryptionkey@0"
I20260806 17:07:06.994155 32713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:07:07.007006 32713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:07:07.010463 32713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:07:07.010699 32713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:07:07.019191 32713 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.125:36099
I20260806 17:07:07.019554 312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.125:36099 every 8 connection(s)
I20260806 17:07:07.020372 32713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-1/data/info.pb
I20260806 17:07:07.022555 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 32713
I20260806 17:07:07.022886 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-1/wal/instance
I20260806 17:07:07.023214 9661 external_mini_cluster.cc:1428] Setting key 62d4214d6014f5803791a62d2b49d80c
I20260806 17:07:07.024145 313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:07:07.030223 313 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.111.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.124@KRBTEST.COM" created.
Aug 06 17:07:07 dist-test-slave-69rg krb5kdc[32623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036026, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.125@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:07:07.041886 32706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } attempt: 3
I20260806 17:07:07.048111 32710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.032472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41636 (local address 127.9.111.126:36141)
0806 17:07:07.032621 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.032625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.034067 (+ 1442us) server_negotiation.cc:407] Connection header received
0806 17:07:07.035360 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.035364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.035446 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.035533 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.036893 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.037433 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.038838 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.039096 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.039233 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.042320 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:07.042328 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:07.042333 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:07.042372 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:07.045153 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.045879 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.045885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.045889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.045993 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.046394 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.046398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.046400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.046648 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:07.046765 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:07.047794 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0806 17:07:07.047924 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 17:07:07.048951 316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.033087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:36141 (local address 127.0.0.1:41636)
0806 17:07:07.033844 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.033918 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0806 17:07:07.035073 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:07:07.035730 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:07:07.035738 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:07:07.036101 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:07:07.036730 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:07.036746 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.037589 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.037592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:07.038672 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:07.038687 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.039325 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.039330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:07.039569 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.040331 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:07:07.040349 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:07:07.042077 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:07:07.045308 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:07.045315 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:07.045331 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:07.045702 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:07.046108 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:07.046112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:07.046116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:07.046281 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:07.046787 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:07:07.046796 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:07:07.047218 (+ 422us) client_negotiation.cc:769] Sending connection context
0806 17:07:07.047618 (+ 400us) client_negotiation.cc:240] Negotiation successful
0806 17:07:07.047925 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":64,"thread_start_us":258,"threads_started":1}
Aug 06 17:07:07 dist-test-slave-69rg krb5kdc[32623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036026, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for kudu/127.9.111.125@KRBTEST.COM
I20260806 17:07:07.050671 313 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } has no permanent_uuid. Determining permanent_uuid...
I20260806 17:07:07.054176 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.042444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:36099 (local address 127.0.0.1:39352)
0806 17:07:07.042563 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.042580 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 17:07:07.042662 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:07:07.043668 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:07:07.043671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:07:07.043921 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:07:07.044024 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:07.044030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.045121 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.045126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:07.045998 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:07.046008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.046859 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.046862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:07.046946 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.047649 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:07:07.047667 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:07:07.049948 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:07:07.052117 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:07.052122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:07.052127 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:07.052412 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:07.052886 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:07.052889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:07.052890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:07.052972 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:07.053564 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:07:07.053568 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:07:07.053678 (+ 110us) client_negotiation.cc:769] Sending connection context
0806 17:07:07.053799 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 17:07:07.053979 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260806 17:07:07.054502 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.042671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.9.111.125:36099)
0806 17:07:07.043278 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.043297 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.043321 (+ 24us) server_negotiation.cc:407] Connection header received
0806 17:07:07.043357 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.043361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.043532 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.043682 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.044143 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.044987 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.046359 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.046743 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.046850 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.050086 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:07.050105 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:07.050117 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:07.050141 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:07.051991 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.052637 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.052643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.052645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.052776 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.053079 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.053082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.053084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.053450 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:07.053558 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:07.054059 (+ 501us) server_negotiation.cc:299] Negotiation successful
0806 17:07:07.054287 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":489,"thread_start_us":123,"threads_started":1}
I20260806 17:07:07.056519 32706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } has no permanent_uuid. Determining permanent_uuid...
I20260806 17:07:07.057286 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.057031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:41739 (local address 127.0.0.1:37058)
0806 17:07:07.057156 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.057221 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
Aug 06 17:07:07 dist-test-slave-69rg krb5kdc[32623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036026, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.125@KRBTEST.COM for kudu/127.9.111.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.124.keytab.
Entry for principal kudu/127.9.111.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.124.keytab.
I20260806 17:07:07.058285 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.057925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:41739 (local address 127.0.0.1:37060)
0806 17:07:07.058144 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.058200 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260806 17:07:07.058426 32706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.124:41739: Network error: Client connection negotiation failed: client connection to 127.9.111.124:41739: connect: Connection refused (error 111)
I20260806 17:07:07.059139 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.124:41739
--webserver_interface=127.9.111.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.124.keytab
--principal=kudu/127.9.111.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:43645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:36141,127.9.111.125:36099,127.9.111.124:41739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:07:07.061717 316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.051257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:36099 (local address 127.0.0.1:39354)
0806 17:07:07.051392 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.051410 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 17:07:07.051490 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:07:07.052628 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:07:07.052631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:07:07.052848 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:07:07.052973 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:07.052978 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.054078 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.054081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:07.054873 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:07.054892 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.055751 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.055754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:07.055874 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.056474 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:07:07.056486 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:07:07.058001 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:07:07.060086 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:07.060092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:07.060097 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:07.060382 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:07.060667 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:07.060672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:07.060677 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:07.060784 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:07.061292 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:07:07.061294 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:07:07.061380 (+ 86us) client_negotiation.cc:769] Sending connection context
0806 17:07:07.061474 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 17:07:07.061586 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":2}
I20260806 17:07:07.061995 319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.051374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39354 (local address 127.9.111.125:36099)
0806 17:07:07.052254 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.052263 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.052276 (+ 13us) server_negotiation.cc:407] Connection header received
0806 17:07:07.052310 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.052313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.052361 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.052464 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.053089 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.053966 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.054989 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.055625 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.055746 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.058179 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:07.058184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:07.058187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:07.058210 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:07.059967 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.060490 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.060493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.060494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.060570 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.060907 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.060912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.060915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.061187 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:07.061296 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:07.061559 (+ 263us) server_negotiation.cc:299] Negotiation successful
0806 17:07:07.061705 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":807,"thread_start_us":175,"threads_started":1}
I20260806 17:07:07.062727 313 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } has no permanent_uuid. Determining permanent_uuid...
I20260806 17:07:07.063454 316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.063203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:41739 (local address 127.0.0.1:37062)
0806 17:07:07.063322 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.063400 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 17:07:07.063721 32731 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.111.124:41739: connect: Connection refused (error 111)
I20260806 17:07:07.064774 316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.064462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:41739 (local address 127.0.0.1:37074)
0806 17:07:07.064631 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.064686 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260806 17:07:07.065032 313 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.124:41739: Network error: Client connection negotiation failed: client connection to 127.9.111.124:41739: connect: Connection refused (error 111)
I20260806 17:07:07.089638 32706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } attempt: 1
I20260806 17:07:07.090684 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.090335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:41739 (local address 127.0.0.1:37088)
0806 17:07:07.090533 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.090601 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 17:07:07.091446 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.091224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:41739 (local address 127.0.0.1:37102)
0806 17:07:07.091337 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.091391 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 17:07:07.091578 32706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.124:41739: Network error: Client connection negotiation failed: client connection to 127.9.111.124:41739: connect: Connection refused (error 111)
I20260806 17:07:07.114431 313 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } attempt: 1
I20260806 17:07:07.115480 316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.115108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:41739 (local address 127.0.0.1:37112)
0806 17:07:07.115305 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.115367 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 17:07:07.116220 316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.116001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:41739 (local address 127.0.0.1:37124)
0806 17:07:07.116115 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.116168 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 17:07:07.116364 313 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.124:41739: Network error: Client connection negotiation failed: client connection to 127.9.111.124:41739: connect: Connection refused (error 111)
I20260806 17:07:07.166803 32706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } attempt: 2
I20260806 17:07:07.167773 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.167457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:41739 (local address 127.0.0.1:37136)
0806 17:07:07.167622 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.167691 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 17:07:07.168526 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.168304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:41739 (local address 127.0.0.1:37138)
0806 17:07:07.168418 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.168467 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 17:07:07.168654 32706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.124:41739: Network error: Client connection negotiation failed: client connection to 127.9.111.124:41739: connect: Connection refused (error 111)
W20260806 17:07:07.177663 320 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 17:07:07.178212 320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:07:07.178288 320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:07:07.178361 320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:07:07.182785 320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:07:07.182879 320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:07:07.182916 320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:07:07.182973 320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:07:07.183012 320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:07:07.184543 313 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } attempt: 2
I20260806 17:07:07.185434 316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.185120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:41739 (local address 127.0.0.1:37152)
0806 17:07:07.185280 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.185355 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 17:07:07.186372 316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.186034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:41739 (local address 127.0.0.1:37162)
0806 17:07:07.186238 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.186319 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 17:07:07.186574 313 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.124:41739: Network error: Client connection negotiation failed: client connection to 127.9.111.124:41739: connect: Connection refused (error 111)
I20260806 17:07:07.189075 320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:43645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:36141,127.9.111.125:36099,127.9.111.124:41739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.124.keytab
--principal=kudu/127.9.111.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.124:41739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.111.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:07:07.190322 320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:07:07.191392 320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:07:07.200887 327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:07:07.200914 326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:07:07.200920 329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:07:07.201285 320 server_base.cc:1034] running on GCE node
I20260806 17:07:07.201942 320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:07:07.203166 320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:07:07.204341 320 hybrid_clock.cc:648] HybridClock initialized: now 1786036027204308 us; error 50 us; skew 500 ppm
Aug 06 17:07:07 dist-test-slave-69rg krb5kdc[32623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036027, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:07:07.207552 320 init.cc:377] Logged in from keytab as kudu/127.9.111.124@KRBTEST.COM (short username kudu)
I20260806 17:07:07.208595 320 webserver.cc:460] Webserver started at http://127.9.111.124:32983/ using document root <none> and password file <none>
I20260806 17:07:07.209178 320 fs_manager.cc:359] Metadata directory not provided
I20260806 17:07:07.209241 320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:07:07.209489 320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:07:07.211702 320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-2/data/instance:
uuid: "2d4ccad2165c48289a740a7c67e72e4e"
format_stamp: "Formatted at 2026-08-06 17:07:07 on dist-test-slave-69rg"
server_key: "7072c446fca685714f39301f7d50d794"
server_key_iv: "5bb47324c19264f5767e0941944d2c3a"
server_key_version: "encryptionkey@0"
I20260806 17:07:07.212311 320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-2/wal/instance:
uuid: "2d4ccad2165c48289a740a7c67e72e4e"
format_stamp: "Formatted at 2026-08-06 17:07:07 on dist-test-slave-69rg"
server_key: "7072c446fca685714f39301f7d50d794"
server_key_iv: "5bb47324c19264f5767e0941944d2c3a"
server_key_version: "encryptionkey@0"
I20260806 17:07:07.216166 320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 17:07:07.218967 336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:07:07.220415 320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 17:07:07.220590 320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-2/wal
uuid: "2d4ccad2165c48289a740a7c67e72e4e"
format_stamp: "Formatted at 2026-08-06 17:07:07 on dist-test-slave-69rg"
server_key: "7072c446fca685714f39301f7d50d794"
server_key_iv: "5bb47324c19264f5767e0941944d2c3a"
server_key_version: "encryptionkey@0"
I20260806 17:07:07.220749 320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:07:07.254969 320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:07:07.258776 320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:07:07.259019 320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:07:07.266531 320 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.124:41739
I20260806 17:07:07.266865 390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.124:41739 every 8 connection(s)
I20260806 17:07:07.267599 320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-2/data/info.pb
I20260806 17:07:07.267834 32706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } attempt: 3
I20260806 17:07:07.268504 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 320
I20260806 17:07:07.268646 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786035732873553-9661-0/minicluster-data/master-2/wal/instance
I20260806 17:07:07.268997 9661 external_mini_cluster.cc:1428] Setting key 5a58ee6cd68caf5b65131a35577afdbe
I20260806 17:07:07.271513 391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:07:07.276789 391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 17:07:07 dist-test-slave-69rg krb5kdc[32623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036026, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for kudu/127.9.111.124@KRBTEST.COM
I20260806 17:07:07.277765 313 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } attempt: 3
Aug 06 17:07:07 dist-test-slave-69rg krb5kdc[32623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.124@KRBTEST.COM
I20260806 17:07:07.284148 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.268578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:41739 (local address 127.0.0.1:37166)
0806 17:07:07.268807 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.268825 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 17:07:07.268928 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:07:07.271865 (+ 2937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:07:07.271870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:07:07.272113 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:07:07.272256 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:07.272266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.273928 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.273932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:07.274760 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:07.274773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.275693 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.275698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:07.275796 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.276422 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:07:07.276439 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:07:07.278278 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:07:07.281645 (+ 3367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:07.281649 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:07.281652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:07.281980 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:07.282453 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:07.282457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:07.282459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:07.282564 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:07.283461 (+ 897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:07:07.283465 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:07:07.283623 (+ 158us) client_negotiation.cc:769] Sending connection context
0806 17:07:07.283747 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 17:07:07.283918 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
Aug 06 17:07:07 dist-test-slave-69rg krb5kdc[32623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036026, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.125@KRBTEST.COM for kudu/127.9.111.124@KRBTEST.COM
I20260806 17:07:07.286134 393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.269401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37166 (local address 127.9.111.124:41739)
0806 17:07:07.270064 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.270084 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.270143 (+ 59us) server_negotiation.cc:407] Connection header received
0806 17:07:07.271085 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.271115 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.271504 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.272410 (+ 906us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.272545 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.273719 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.274919 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.275522 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.276105 (+ 583us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.278526 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:07.278534 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:07.278551 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:07.278586 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:07.281496 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.282144 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.282151 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.282157 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.282335 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.282685 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.282689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.282691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.283321 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:07.283474 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:07.284481 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0806 17:07:07.284968 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":241,"server-negotiator.queue_time_us":505,"spinlock_wait_cycles":13312,"thread_start_us":158,"threads_started":1}
I20260806 17:07:07.286131 394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.269596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.9.111.124:41739)
0806 17:07:07.270093 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.270103 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.270350 (+ 247us) server_negotiation.cc:407] Connection header received
0806 17:07:07.271091 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.271114 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.271499 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.271840 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.272723 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.273672 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.274928 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.275511 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.275692 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.279821 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:07.279824 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:07.279827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:07.279854 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:07.282014 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.282531 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.282534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.282536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.282614 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.282944 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.282947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.282949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.283499 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:07.284198 (+ 699us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:07.284521 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 17:07:07.284865 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":415,"spinlock_wait_cycles":768,"thread_start_us":108,"threads_started":1}
Aug 06 17:07:07 dist-test-slave-69rg krb5kdc[32623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036027, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.124@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:07:07.291074 316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.278308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:41739 (local address 127.0.0.1:37188)
0806 17:07:07.278581 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.278597 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:07:07.278689 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:07:07.279426 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:07:07.279430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:07:07.279703 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:07:07.279863 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:07.279876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.281203 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.281207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:07.282179 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:07.282191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.282887 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.282891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:07.282986 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.284509 (+ 1523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:07:07.284526 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:07:07.286269 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:07:07.289220 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:07.289223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:07.289226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:07.289497 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:07.289879 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:07.289882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:07.289884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:07.289976 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:07.290511 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:07:07.290514 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:07:07.290627 (+ 113us) client_negotiation.cc:769] Sending connection context
0806 17:07:07.290744 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 17:07:07.290901 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
I20260806 17:07:07.291090 396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.278572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37188 (local address 127.9.111.124:41739)
0806 17:07:07.279079 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.279088 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.279133 (+ 45us) server_negotiation.cc:407] Connection header received
0806 17:07:07.279176 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.279179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.279245 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.280006 (+ 761us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.280137 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.281030 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.282335 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.282728 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.283233 (+ 505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.286873 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:07.286877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:07.286880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:07.286907 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:07.289080 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.289632 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.289636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.289638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.289740 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.290091 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.290095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.290097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.290361 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:07.290515 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:07.290817 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 17:07:07.290931 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":116,"threads_started":1}
I20260806 17:07:07.291386 32706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466: Bootstrap starting.
I20260806 17:07:07.291736 397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.279229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:36141 (local address 127.0.0.1:41650)
0806 17:07:07.280239 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.280318 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0806 17:07:07.280492 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:07:07.280933 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:07:07.280941 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:07:07.281563 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:07:07.281739 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:07.281748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.282820 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.282824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:07.283928 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:07.283941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.284750 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.284753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:07.284873 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.285384 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:07:07.285400 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:07:07.287577 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:07:07.289779 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:07.289788 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:07.289794 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:07.290119 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:07.290427 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:07.290432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:07.290434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:07.290519 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:07.291059 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:07:07.291065 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:07:07.291193 (+ 128us) client_negotiation.cc:769] Sending connection context
0806 17:07:07.291349 (+ 156us) client_negotiation.cc:240] Negotiation successful
0806 17:07:07.291549 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":107,"threads_started":1}
I20260806 17:07:07.291759 32710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.279320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.9.111.126:36141)
0806 17:07:07.279466 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.279473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.280395 (+ 922us) server_negotiation.cc:407] Connection header received
0806 17:07:07.280621 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.280626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.280682 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.280742 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.281931 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.282670 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.284089 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.284412 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.284486 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.287746 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:07.287750 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:07.287752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:07.287779 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:07.289652 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.290235 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.290238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.290240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.290334 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.290622 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.290626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.290627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.290892 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:07.290997 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:07.291468 (+ 471us) server_negotiation.cc:299] Negotiation successful
0806 17:07:07.291607 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 17:07:07.292897 391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } has no permanent_uuid. Determining permanent_uuid...
I20260806 17:07:07.294955 32706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466: Neither blocks nor log segments found. Creating new log.
I20260806 17:07:07.295198 313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7: Bootstrap starting.
I20260806 17:07:07.296089 32706 log.cc:824] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466: Log is configured to *not* fsync() on all Append() calls
I20260806 17:07:07.297988 313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7: Neither blocks nor log segments found. Creating new log.
I20260806 17:07:07.299054 313 log.cc:824] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7: Log is configured to *not* fsync() on all Append() calls
I20260806 17:07:07.299125 32706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466: No bootstrap required, opened a new log
Aug 06 17:07:07 dist-test-slave-69rg krb5kdc[32623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036027, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.124@KRBTEST.COM for kudu/127.9.111.125@KRBTEST.COM
Aug 06 17:07:07 dist-test-slave-69rg krb5kdc[32623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:07:07.303594 313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7: No bootstrap required, opened a new log
I20260806 17:07:07.304217 397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.293429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:36099 (local address 127.0.0.1:39364)
0806 17:07:07.293687 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.293705 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 17:07:07.293798 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:07:07.294210 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:07:07.294213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:07:07.294401 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:07:07.294499 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:07.294506 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.295711 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.295714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:07.296589 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:07.296611 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.297507 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.297511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:07.297606 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.298227 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:07:07.298243 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:07:07.299945 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:07:07.302359 (+ 2414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:07.302365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:07.302370 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:07.302664 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:07.303012 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:07.303015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:07.303017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:07.303112 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:07.303676 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:07:07.303679 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:07:07.303778 (+ 99us) client_negotiation.cc:769] Sending connection context
0806 17:07:07.303881 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 17:07:07.304033 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260806 17:07:07.304234 32706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37846176c33848e1b5cb06ae8fc76466" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } } peers { permanent_uuid: "8cfb28f0db9f41e7a6728a07514be3c7" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } } peers { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } }
I20260806 17:07:07.304479 32706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37846176c33848e1b5cb06ae8fc76466, State: Initialized, Role: FOLLOWER
I20260806 17:07:07.305063 32710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.292022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.9.111.126:36141)
0806 17:07:07.292146 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.292150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.294051 (+ 1901us) server_negotiation.cc:407] Connection header received
0806 17:07:07.294264 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.294267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.294313 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.294360 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.295269 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.295951 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.297074 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.297380 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.297454 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.301113 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:07.301117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:07.301120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:07.301148 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:07.302958 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.303490 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.303494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.303496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.303584 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.303916 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.303920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.303921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.304144 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:07.304327 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:07.304707 (+ 380us) server_negotiation.cc:299] Negotiation successful
0806 17:07:07.304838 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 17:07:07.305183 319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.293547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39364 (local address 127.9.111.125:36099)
0806 17:07:07.293686 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.293691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.293752 (+ 61us) server_negotiation.cc:407] Connection header received
0806 17:07:07.293946 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.293950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.294003 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.294057 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.294660 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.295508 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.296764 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.297253 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.297373 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.300122 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:07.300129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:07.300133 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:07.300162 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:07.302109 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.302801 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.302805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.302807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.302900 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.303233 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.303236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.303238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.303531 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:07.303627 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:07.304869 (+ 1242us) server_negotiation.cc:299] Negotiation successful
0806 17:07:07.305010 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 17:07:07.304855 32706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37846176c33848e1b5cb06ae8fc76466" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } } peers { permanent_uuid: "8cfb28f0db9f41e7a6728a07514be3c7" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } } peers { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } }
I20260806 17:07:07.305930 391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } has no permanent_uuid. Determining permanent_uuid...
I20260806 17:07:07.307847 32706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:07:07.308171 313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37846176c33848e1b5cb06ae8fc76466" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } } peers { permanent_uuid: "8cfb28f0db9f41e7a6728a07514be3c7" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } } peers { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } }
I20260806 17:07:07.308501 313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cfb28f0db9f41e7a6728a07514be3c7, State: Initialized, Role: FOLLOWER
I20260806 17:07:07.308904 313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37846176c33848e1b5cb06ae8fc76466" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } } peers { permanent_uuid: "8cfb28f0db9f41e7a6728a07514be3c7" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } } peers { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } }
I20260806 17:07:07.310758 313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7 [sys.catalog]: configured and running, proceeding with master startup.
Aug 06 17:07:07 dist-test-slave-69rg krb5kdc[32623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036027, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.124@KRBTEST.COM for kudu/127.9.111.124@KRBTEST.COM
Aug 06 17:07:07 dist-test-slave-69rg krb5kdc[32623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.125@KRBTEST.COM
I20260806 17:07:07.310606 400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37846176c33848e1b5cb06ae8fc76466" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } } peers { permanent_uuid: "8cfb28f0db9f41e7a6728a07514be3c7" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } } peers { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } } }
I20260806 17:07:07.314999 400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:07:07.315794 397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.306433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:41739 (local address 127.0.0.1:37198)
0806 17:07:07.306714 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:07.306730 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 17:07:07.306805 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:07:07.307057 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:07:07.307060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:07:07.307239 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:07:07.307341 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:07.307349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.308328 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.308332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:07.309088 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:07.309098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.309665 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.309668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:07.309738 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.310310 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:07:07.310326 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:07:07.312005 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:07:07.314144 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:07.314147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:07.314150 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:07.314454 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:07.314737 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:07.314739 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:07.314741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:07.314827 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:07.315282 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:07:07.315284 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:07:07.315370 (+ 86us) client_negotiation.cc:769] Sending connection context
0806 17:07:07.315467 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 17:07:07.315622 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260806 17:07:07.316628 396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.306717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.9.111.124:41739)
0806 17:07:07.306829 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.306834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.306846 (+ 12us) server_negotiation.cc:407] Connection header received
0806 17:07:07.306885 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.306889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.306930 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.307023 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.307463 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.308193 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.309222 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.309549 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.309996 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.312162 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:07.312167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:07.312170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:07.312192 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:07.314031 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.314553 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.314556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.314560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.314644 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.314925 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.314927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.314929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.315141 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:07.315233 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:07.316333 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0806 17:07:07.316467 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 17:07:07.316550 406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37846176c33848e1b5cb06ae8fc76466" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } } peers { permanent_uuid: "8cfb28f0db9f41e7a6728a07514be3c7" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } } peers { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } } }
I20260806 17:07:07.316682 406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:07:07.318185 319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.306765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39374 (local address 127.9.111.125:36099)
0806 17:07:07.306857 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.306863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.307019 (+ 156us) server_negotiation.cc:407] Connection header received
0806 17:07:07.307213 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.307216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.307257 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.307305 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.308196 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.308913 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.310434 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.310772 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.310878 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.314181 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:07.314185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:07.314188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:07.314218 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:07.315778 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.316325 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.316329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.316330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.316418 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.316727 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.316730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.316732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.316940 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:07.317147 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:07.317871 (+ 724us) server_negotiation.cc:299] Negotiation successful
0806 17:07:07.318009 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260806 17:07:07.319933 412 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:07:07.320087 412 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:07:07.321677 391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e: Bootstrap starting.
I20260806 17:07:07.322511 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:07:07.319676 32637 master_runner.cc:186] Error getting master registration for 127.9.111.124:41739: OK, instance_id { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" instance_seqno: 1786036027264386 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 17:07:07.324090 32637 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 17:07:07.323984 32679 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "37846176c33848e1b5cb06ae8fc76466"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41656
I20260806 17:07:07.324291 32679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 17:07:07.324406 32679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:07:07.321149 32713 master_runner.cc:186] Error getting master registration for 127.9.111.124:41739: OK, instance_id { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" instance_seqno: 1786036027264386 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 17:07:07.324841 32713 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 17:07:07.325065 391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e: Neither blocks nor log segments found. Creating new log.
I20260806 17:07:07.325965 32679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37846176c33848e1b5cb06ae8fc76466" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } } peers { permanent_uuid: "8cfb28f0db9f41e7a6728a07514be3c7" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } } peers { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } }
I20260806 17:07:07.326030 391 log.cc:824] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e: Log is configured to *not* fsync() on all Append() calls
I20260806 17:07:07.326980 32679 leader_election.cc:288] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [CANDIDATE]: Term 1 election: Requested vote from peers 8cfb28f0db9f41e7a6728a07514be3c7 (127.9.111.125:36099), 2d4ccad2165c48289a740a7c67e72e4e (127.9.111.124:41739)
I20260806 17:07:07.327971 32755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37846176c33848e1b5cb06ae8fc76466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8cfb28f0db9f41e7a6728a07514be3c7"
I20260806 17:07:07.328243 32755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:07:07.328399 391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e: No bootstrap required, opened a new log
I20260806 17:07:07.329698 32755 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37846176c33848e1b5cb06ae8fc76466 in term 1.
W20260806 17:07:07.329871 423 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:07:07.329957 423 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:07:07.330271 32655 leader_election.cc:302] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37846176c33848e1b5cb06ae8fc76466, 8cfb28f0db9f41e7a6728a07514be3c7; no voters:
I20260806 17:07:07.330714 400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:07:07.330971 400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [term 1 LEADER]: Becoming Leader. State: Replica: 37846176c33848e1b5cb06ae8fc76466, State: Running, Role: LEADER
I20260806 17:07:07.331374 400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37846176c33848e1b5cb06ae8fc76466" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } } peers { permanent_uuid: "8cfb28f0db9f41e7a6728a07514be3c7" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } } peers { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } }
I20260806 17:07:07.333420 391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37846176c33848e1b5cb06ae8fc76466" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } } peers { permanent_uuid: "8cfb28f0db9f41e7a6728a07514be3c7" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } } peers { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } }
I20260806 17:07:07.333774 391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d4ccad2165c48289a740a7c67e72e4e, State: Initialized, Role: FOLLOWER
I20260806 17:07:07.334199 391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37846176c33848e1b5cb06ae8fc76466" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } } peers { permanent_uuid: "8cfb28f0db9f41e7a6728a07514be3c7" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } } peers { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } }
I20260806 17:07:07.334651 431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37846176c33848e1b5cb06ae8fc76466. Latest consensus state: current_term: 1 leader_uuid: "37846176c33848e1b5cb06ae8fc76466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37846176c33848e1b5cb06ae8fc76466" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } } peers { permanent_uuid: "8cfb28f0db9f41e7a6728a07514be3c7" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } } peers { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } } }
I20260806 17:07:07.335018 431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [sys.catalog]: This master's current role is: LEADER
I20260806 17:07:07.335736 391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:07:07.335810 364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37846176c33848e1b5cb06ae8fc76466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2d4ccad2165c48289a740a7c67e72e4e"
I20260806 17:07:07.336066 364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:07:07.339073 364 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37846176c33848e1b5cb06ae8fc76466 in term 1.
I20260806 17:07:07.341595 437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:07:07.342393 434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37846176c33848e1b5cb06ae8fc76466" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } } peers { permanent_uuid: "8cfb28f0db9f41e7a6728a07514be3c7" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } } peers { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } } }
I20260806 17:07:07.342651 434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:07:07.342856 437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 17:07:07.350626 319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.333763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39382 (local address 127.9.111.125:36099)
0806 17:07:07.333964 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.333971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.335376 (+ 1405us) server_negotiation.cc:407] Connection header received
0806 17:07:07.335561 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.335564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.335624 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.335702 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.338828 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.339625 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.343106 (+ 3481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.343430 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.343514 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.346663 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:07.346667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:07.346670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:07.346697 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:07.348471 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.349024 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.349030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.349032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.349127 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.349471 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.349474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.349476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.349740 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:07.349914 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:07.350298 (+ 384us) server_negotiation.cc:299] Negotiation successful
0806 17:07:07.350437 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260806 17:07:07.352332 32710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.330500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.9.111.126:36141)
0806 17:07:07.330645 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.330650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.337028 (+ 6378us) server_negotiation.cc:407] Connection header received
0806 17:07:07.337237 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.337241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.337298 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.337372 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.341292 (+ 3920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.341996 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.344257 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.344577 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.344654 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.348412 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:07.348416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:07.348420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:07.348455 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:07.350221 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.350709 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.350714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.350715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.350809 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.351408 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.351413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.351414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.351620 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:07.351714 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:07.352013 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 17:07:07.352157 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260806 17:07:07.354259 448 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:07:07.354378 448 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:07:07.354974 396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:07.334401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37212 (local address 127.9.111.124:41739)
0806 17:07:07.334535 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.334540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.338693 (+ 4153us) server_negotiation.cc:407] Connection header received
0806 17:07:07.338781 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.338784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.338838 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.338931 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.342909 (+ 3978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.343726 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.347254 (+ 3528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:07.347616 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:07.347694 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:07.350082 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:07.350086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:07.350088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:07.350122 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:07.351768 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.353542 (+ 1774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.353546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.353548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.353639 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:07.354017 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:07.354021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:07.354023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:07.354257 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:07.354394 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:07.354704 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 17:07:07.354825 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 17:07:07.365767 364 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e [term 1 FOLLOWER]: Refusing update from remote peer 37846176c33848e1b5cb06ae8fc76466: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:07:07.366637 431 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:07:07.373601 434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37846176c33848e1b5cb06ae8fc76466. Latest consensus state: current_term: 1 leader_uuid: "37846176c33848e1b5cb06ae8fc76466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37846176c33848e1b5cb06ae8fc76466" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } } peers { permanent_uuid: "8cfb28f0db9f41e7a6728a07514be3c7" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } } peers { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } } }
I20260806 17:07:07.373768 434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:07:07.375524 32755 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7 [term 1 FOLLOWER]: Refusing update from remote peer 37846176c33848e1b5cb06ae8fc76466: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:07:07.378607 431 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cfb28f0db9f41e7a6728a07514be3c7" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:07:07.383313 406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37846176c33848e1b5cb06ae8fc76466. Latest consensus state: current_term: 1 leader_uuid: "37846176c33848e1b5cb06ae8fc76466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37846176c33848e1b5cb06ae8fc76466" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } } peers { permanent_uuid: "8cfb28f0db9f41e7a6728a07514be3c7" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } } peers { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } } }
I20260806 17:07:07.383458 406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:07:07.390386 431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37846176c33848e1b5cb06ae8fc76466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37846176c33848e1b5cb06ae8fc76466" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } } peers { permanent_uuid: "8cfb28f0db9f41e7a6728a07514be3c7" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } } peers { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } } }
I20260806 17:07:07.390579 431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [sys.catalog]: This master's current role is: LEADER
I20260806 17:07:07.392246 434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37846176c33848e1b5cb06ae8fc76466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37846176c33848e1b5cb06ae8fc76466" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } } peers { permanent_uuid: "8cfb28f0db9f41e7a6728a07514be3c7" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } } peers { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } } }
I20260806 17:07:07.392398 434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:07:07.395229 437 catalog_manager.cc:1347] Generated new cluster ID: 98ae756e1b614391bd8d35ed8eb10aa4
I20260806 17:07:07.395357 437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:07:07.396127 400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37846176c33848e1b5cb06ae8fc76466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37846176c33848e1b5cb06ae8fc76466" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } } peers { permanent_uuid: "8cfb28f0db9f41e7a6728a07514be3c7" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } } peers { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } } }
I20260806 17:07:07.396287 400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466 [sys.catalog]: This master's current role is: LEADER
I20260806 17:07:07.398006 406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37846176c33848e1b5cb06ae8fc76466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37846176c33848e1b5cb06ae8fc76466" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 36141 } } peers { permanent_uuid: "8cfb28f0db9f41e7a6728a07514be3c7" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 36099 } } peers { permanent_uuid: "2d4ccad2165c48289a740a7c67e72e4e" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 41739 } } }
I20260806 17:07:07.398185 406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:07:07.398715 449 mvcc.cc:205] Tried to move back new op lower bound from 7315603568081494016 to 7315603567953174528. Current Snapshot: MvccSnapshot[applied={T|T < 7315603568081494016 or (T in {7315603568081494016})}]
I20260806 17:07:07.398979 453 mvcc.cc:205] Tried to move back new op lower bound from 7315603568081494016 to 7315603567953174528. Current Snapshot: MvccSnapshot[applied={T|T < 7315603568081494016}]
I20260806 17:07:07.399519 451 mvcc.cc:205] Tried to move back new op lower bound from 7315603568081494016 to 7315603567953174528. Current Snapshot: MvccSnapshot[applied={T|T < 7315603568081494016}]
I20260806 17:07:07.439762 437 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:07:07.441212 437 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:07:07.456120 437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37846176c33848e1b5cb06ae8fc76466: Generated new TSK 0
I20260806 17:07:07.457134 437 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 17:07:07.491819 319 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:07.488391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39394 (local address 127.9.111.125:36099)
0806 17:07:07.488532 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.488537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.490602 (+ 2065us) server_negotiation.cc:407] Connection header received
0806 17:07:07.490862 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.490867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.490921 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.490979 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.491672 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39394: BlockingRecv error: recv got EOF from 127.0.0.1:39394 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 17:07:07.493440 396 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:07.489185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37226 (local address 127.9.111.124:41739)
0806 17:07:07.489339 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.489344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.492479 (+ 3135us) server_negotiation.cc:407] Connection header received
0806 17:07:07.492651 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.492654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.492718 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.492771 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.493307 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37226: BlockingRecv error: recv got EOF from 127.0.0.1:37226 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 17:07:07.495235 32710 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:07.489890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41670 (local address 127.9.111.126:36141)
0806 17:07:07.490011 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.490015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.494112 (+ 4097us) server_negotiation.cc:407] Connection header received
0806 17:07:07.494330 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.494334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.494387 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.494444 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.495107 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41670: BlockingRecv error: recv got EOF from 127.0.0.1:41670 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 17:07:07.498945 319 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:07.497012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39410 (local address 127.9.111.125:36099)
0806 17:07:07.497142 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.497147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.497699 (+ 552us) server_negotiation.cc:407] Connection header received
0806 17:07:07.497985 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.497989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.498072 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.498182 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.498821 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39410: BlockingRecv error: recv got EOF from 127.0.0.1:39410 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 17:07:07.504616 319 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:07.502847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.9.111.125:36099)
0806 17:07:07.502970 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.502975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.503588 (+ 613us) server_negotiation.cc:407] Connection header received
0806 17:07:07.503799 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.503803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.503858 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.503913 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.504495 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39426: BlockingRecv error: recv got EOF from 127.0.0.1:39426 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 17:07:07.512106 319 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:07.509339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39438 (local address 127.9.111.125:36099)
0806 17:07:07.509460 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.509465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.510087 (+ 622us) server_negotiation.cc:407] Connection header received
0806 17:07:07.510420 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.510424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.510483 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.510542 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.511981 (+ 1439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39438: BlockingRecv error: recv got EOF from 127.0.0.1:39438 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 17:07:07.520771 319 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:07.518790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39442 (local address 127.9.111.125:36099)
0806 17:07:07.518951 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.518956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.519623 (+ 667us) server_negotiation.cc:407] Connection header received
0806 17:07:07.519863 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.519867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.519928 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.519991 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.520640 (+ 649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39442: BlockingRecv error: recv got EOF from 127.0.0.1:39442 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260806 17:07:07.533785 319 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:07.531761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39452 (local address 127.9.111.125:36099)
0806 17:07:07.531904 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.531910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.532647 (+ 737us) server_negotiation.cc:407] Connection header received
0806 17:07:07.532890 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.532893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.532955 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.533015 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.533655 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39452: BlockingRecv error: recv got EOF from 127.0.0.1:39452 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 17:07:07.554982 319 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:07.552808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39462 (local address 127.9.111.125:36099)
0806 17:07:07.552956 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.552962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.553670 (+ 708us) server_negotiation.cc:407] Connection header received
0806 17:07:07.553994 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.553998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.554072 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.554163 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.554852 (+ 689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39462: BlockingRecv error: recv got EOF from 127.0.0.1:39462 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 17:07:07.591274 319 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:07.589925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39478 (local address 127.9.111.125:36099)
0806 17:07:07.590166 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.590173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.590258 (+ 85us) server_negotiation.cc:407] Connection header received
0806 17:07:07.590459 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.590463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.590519 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.590618 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.591147 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39478: BlockingRecv error: recv got EOF from 127.0.0.1:39478 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260806 17:07:07.659178 319 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:07.657778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39482 (local address 127.9.111.125:36099)
0806 17:07:07.657992 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.657998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.658163 (+ 165us) server_negotiation.cc:407] Connection header received
0806 17:07:07.658340 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.658344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.658406 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.658510 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.659024 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39482: BlockingRecv error: recv got EOF from 127.0.0.1:39482 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260806 17:07:07.791172 319 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:07.789745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39484 (local address 127.9.111.125:36099)
0806 17:07:07.789931 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:07.789936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:07.790172 (+ 236us) server_negotiation.cc:407] Connection header received
0806 17:07:07.790378 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:07.790381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:07.790431 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:07.790526 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:07.791023 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39484: BlockingRecv error: recv got EOF from 127.0.0.1:39484 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260806 17:07:08.051334 319 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:08.049895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39488 (local address 127.9.111.125:36099)
0806 17:07:08.050091 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:08.050096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:08.050218 (+ 122us) server_negotiation.cc:407] Connection header received
0806 17:07:08.050451 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:08.050455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:08.050514 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:08.050620 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:08.051126 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39488: BlockingRecv error: recv got EOF from 127.0.0.1:39488 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 17:07:08.331418 423 catalog_manager.cc:1259] Loaded cluster ID: 98ae756e1b614391bd8d35ed8eb10aa4
I20260806 17:07:08.331517 423 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7: loading cluster ID for follower catalog manager: success
I20260806 17:07:08.333302 423 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7: acquiring CA information for follower catalog manager: success
I20260806 17:07:08.334736 423 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8cfb28f0db9f41e7a6728a07514be3c7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 17:07:08.356002 448 catalog_manager.cc:1259] Loaded cluster ID: 98ae756e1b614391bd8d35ed8eb10aa4
I20260806 17:07:08.356091 448 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e: loading cluster ID for follower catalog manager: success
I20260806 17:07:08.358076 448 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e: acquiring CA information for follower catalog manager: success
I20260806 17:07:08.359433 448 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2d4ccad2165c48289a740a7c67e72e4e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 17:07:08.569798 523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:08.565627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39496 (local address 127.9.111.125:36099)
0806 17:07:08.565944 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:08.565951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:08.565969 (+ 18us) server_negotiation.cc:407] Connection header received
0806 17:07:08.566154 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:08.566158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:08.566347 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:08.566474 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:07:08.566745 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:08.567301 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:08.568185 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:08.568437 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:08.568552 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:08.568911 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:07:08.569364 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:07:08.569505 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:08.569566 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 17:07:08.569626 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":87,"threads_started":1}
I20260806 17:07:08.577379 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:08.573447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.9.111.124:41739)
0806 17:07:08.573772 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:08.573779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:08.573794 (+ 15us) server_negotiation.cc:407] Connection header received
0806 17:07:08.573921 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:08.573924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:08.574040 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:08.574177 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:07:08.574461 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:08.574984 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:08.575862 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:08.576084 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:08.576205 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:08.576531 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:07:08.576969 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:07:08.577107 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:08.577163 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 17:07:08.577219 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":98,"threads_started":1}
I20260806 17:07:08.585198 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:08.580620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41678 (local address 127.9.111.126:36141)
0806 17:07:08.581220 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:08.581223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:08.581234 (+ 11us) server_negotiation.cc:407] Connection header received
0806 17:07:08.581317 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:08.581319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:08.581432 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:08.581560 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:07:08.581794 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:08.582688 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:08.583639 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:08.583896 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:08.584010 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:08.584392 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:07:08.584743 (+ 351us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:07:08.584843 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:08.584951 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 17:07:08.585013 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":66,"threads_started":1}
I20260806 17:07:08.585398 523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:08.580704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39508 (local address 127.9.111.125:36099)
0806 17:07:08.580843 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:08.580847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:08.581062 (+ 215us) server_negotiation.cc:407] Connection header received
0806 17:07:08.581880 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:08.581887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:08.581952 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:08.582021 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:07:08.582239 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:08.582773 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:08.583662 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:08.584012 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:08.584111 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:08.584433 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:07:08.584521 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:07:08.584593 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:08.585186 (+ 593us) server_negotiation.cc:299] Negotiation successful
0806 17:07:08.585254 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 17:07:08.586282 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:08.581061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.9.111.124:41739)
0806 17:07:08.581169 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:08.581172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:08.581381 (+ 209us) server_negotiation.cc:407] Connection header received
0806 17:07:08.581525 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:08.581528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:08.581570 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:08.581661 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:07:08.581956 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:08.582687 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:08.584631 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:08.584873 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:08.584990 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:08.585427 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:07:08.585540 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:07:08.585592 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:08.586101 (+ 509us) server_negotiation.cc:299] Negotiation successful
0806 17:07:08.586162 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 17:07:08.588846 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 32637
I20260806 17:07:08.596069 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 32713
I20260806 17:07:08.603633 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 320
2026-08-06T17:07:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3230 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 17:07:08 dist-test-slave-69rg krb5kdc[540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 17:07:08 dist-test-slave-69rg krb5kdc[540](info): set up 2 sockets
Aug 06 17:07:08 dist-test-slave-69rg krb5kdc[540](info): commencing operation
krb5kdc: starting...
W20260806 17:07:09.691026 9661 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/test-user.keytab.
Aug 06 17:07:09 dist-test-slave-69rg krb5kdc[540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T17:07:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T17:07:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.111.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
Entry for principal kudu/127.9.111.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab.
I20260806 17:07:09.838521 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:41975
--webserver_interface=127.9.111.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:41975,127.9.111.125:32979,127.9.111.124:34763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 17:07:09.957163 553 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 17:07:09.957588 553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:07:09.957679 553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:07:09.957763 553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:07:09.962270 553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:07:09.962409 553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:07:09.962445 553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:07:09.962478 553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:07:09.962513 553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:07:09.968698 553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:41975,127.9.111.125:32979,127.9.111.124:34763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.126.keytab
--principal=kudu/127.9.111.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.126:41975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.111.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:07:09.970098 553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:07:09.971261 553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:07:09.981614 559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:07:09.981729 561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:07:09.981753 558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:07:09.982246 553 server_base.cc:1034] running on GCE node
I20260806 17:07:09.982846 553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:07:09.984481 553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:07:09.985730 553 hybrid_clock.cc:648] HybridClock initialized: now 1786036029985693 us; error 52 us; skew 500 ppm
Aug 06 17:07:09 dist-test-slave-69rg krb5kdc[540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036029, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:07:09.994922 553 init.cc:377] Logged in from keytab as kudu/127.9.111.126@KRBTEST.COM (short username kudu)
I20260806 17:07:09.996320 553 webserver.cc:460] Webserver started at http://127.9.111.126:33877/ using document root <none> and password file <none>
I20260806 17:07:09.997110 553 fs_manager.cc:359] Metadata directory not provided
I20260806 17:07:09.997274 553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:07:09.997555 553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:07:10.000669 553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-0/data/instance:
uuid: "39ab68f9d2e3499da77eeee0e4b01f2d"
format_stamp: "Formatted at 2026-08-06 17:07:09 on dist-test-slave-69rg"
server_key: "cd7d86fee0569d2d9a292f017437bab9"
server_key_iv: "a757f30d5218ef385813ebf56d153e9e"
server_key_version: "encryptionkey@0"
I20260806 17:07:10.001416 553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-0/wal/instance:
uuid: "39ab68f9d2e3499da77eeee0e4b01f2d"
format_stamp: "Formatted at 2026-08-06 17:07:09 on dist-test-slave-69rg"
server_key: "cd7d86fee0569d2d9a292f017437bab9"
server_key_iv: "a757f30d5218ef385813ebf56d153e9e"
server_key_version: "encryptionkey@0"
I20260806 17:07:10.006475 553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 17:07:10.009589 568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:07:10.011857 553 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 17:07:10.012156 553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-0/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-0/wal
uuid: "39ab68f9d2e3499da77eeee0e4b01f2d"
format_stamp: "Formatted at 2026-08-06 17:07:09 on dist-test-slave-69rg"
server_key: "cd7d86fee0569d2d9a292f017437bab9"
server_key_iv: "a757f30d5218ef385813ebf56d153e9e"
server_key_version: "encryptionkey@0"
I20260806 17:07:10.012320 553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:07:10.042541 553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:07:10.046723 553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:07:10.046998 553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:07:10.055991 553 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.126:41975
I20260806 17:07:10.056535 621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.126:41975 every 8 connection(s)
I20260806 17:07:10.057498 553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-0/data/info.pb
I20260806 17:07:10.058583 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 553
I20260806 17:07:10.058761 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-0/wal/instance
I20260806 17:07:10.059120 9661 external_mini_cluster.cc:1428] Setting key e757acd4ca7cb707b003052b5e1d9093
I20260806 17:07:10.062216 622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:07:10.068500 622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.111.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.125@KRBTEST.COM" created.
Aug 06 17:07:10 dist-test-slave-69rg krb5kdc[540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036029, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:07:10.092387 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.071458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:41975 (local address 127.0.0.1:53324)
0806 17:07:10.072125 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.072183 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 17:07:10.073510 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:07:10.075224 (+ 1714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:07:10.075235 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:07:10.075659 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:07:10.075992 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:10.076010 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.078002 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.078006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:10.079469 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:10.079482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.081521 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.081525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:10.082010 (+ 485us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.082752 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:07:10.082773 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:07:10.085104 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:07:10.087915 (+ 2811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:10.087925 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:10.087938 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:10.088411 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:10.089159 (+ 748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:10.089164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:10.089166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:10.089318 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:10.090207 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:07:10.090214 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:07:10.090372 (+ 158us) client_negotiation.cc:769] Sending connection context
0806 17:07:10.090650 (+ 278us) client_negotiation.cc:240] Negotiation successful
0806 17:07:10.091099 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":154,"thread_start_us":212,"threads_started":1}
I20260806 17:07:10.092387 626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.072583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.9.111.126:41975)
0806 17:07:10.072937 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.072950 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.073003 (+ 53us) server_negotiation.cc:407] Connection header received
0806 17:07:10.074677 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.074682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.074923 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.075240 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.076184 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.077613 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.079964 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.081203 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.081536 (+ 333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.085309 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:10.085322 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:10.085334 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:10.085358 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:10.087710 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.088670 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.088678 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.088679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.088865 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.089432 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.089435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.089437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.090074 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:10.090235 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:10.090728 (+ 493us) server_negotiation.cc:299] Negotiation successful
0806 17:07:10.091039 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":229,"spinlock_wait_cycles":2304,"thread_start_us":112,"threads_started":1}
I20260806 17:07:10.096413 622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } has no permanent_uuid. Determining permanent_uuid...
I20260806 17:07:10.097764 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.097017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:32979 (local address 127.0.0.1:43532)
0806 17:07:10.097192 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.097642 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260806 17:07:10.098153 572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.111.125:32979: connect: Connection refused (error 111)
I20260806 17:07:10.099763 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.099306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:32979 (local address 127.0.0.1:43546)
0806 17:07:10.099613 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.099684 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225}
W20260806 17:07:10.099943 622 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.125:32979: Network error: Client connection negotiation failed: client connection to 127.9.111.125:32979: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.125.keytab.
Entry for principal kudu/127.9.111.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.125.keytab.
I20260806 17:07:10.109124 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.125:32979
--webserver_interface=127.9.111.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.125.keytab
--principal=kudu/127.9.111.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:41975,127.9.111.125:32979,127.9.111.124:34763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:07:10.149279 622 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } attempt: 1
I20260806 17:07:10.150532 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.150215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:32979 (local address 127.0.0.1:43550)
0806 17:07:10.150379 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.150452 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 17:07:10.151559 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.151236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:32979 (local address 127.0.0.1:43554)
0806 17:07:10.151419 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.151487 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260806 17:07:10.151791 622 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.125:32979: Network error: Client connection negotiation failed: client connection to 127.9.111.125:32979: connect: Connection refused (error 111)
I20260806 17:07:10.220006 622 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } attempt: 2
I20260806 17:07:10.220992 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.220661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:32979 (local address 127.0.0.1:43558)
0806 17:07:10.220838 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.220911 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 17:07:10.222038 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.221713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:32979 (local address 127.0.0.1:43566)
0806 17:07:10.221904 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.221966 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260806 17:07:10.222257 622 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.125:32979: Network error: Client connection negotiation failed: client connection to 127.9.111.125:32979: connect: Connection refused (error 111)
W20260806 17:07:10.245992 629 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 17:07:10.246430 629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:07:10.246495 629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:07:10.246575 629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:07:10.251132 629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:07:10.251255 629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:07:10.251295 629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:07:10.251353 629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:07:10.251390 629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:07:10.257184 629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:41975,127.9.111.125:32979,127.9.111.124:34763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.125.keytab
--principal=kudu/127.9.111.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.125:32979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.111.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:07:10.258543 629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:07:10.259753 629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 17:07:10.269488 634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:07:10.269539 635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:07:10.269528 637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:07:10.269888 629 server_base.cc:1034] running on GCE node
I20260806 17:07:10.270819 629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:07:10.272383 629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:07:10.273587 629 hybrid_clock.cc:648] HybridClock initialized: now 1786036030273547 us; error 60 us; skew 500 ppm
Aug 06 17:07:10 dist-test-slave-69rg krb5kdc[540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036030, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:07:10.277158 629 init.cc:377] Logged in from keytab as kudu/127.9.111.125@KRBTEST.COM (short username kudu)
I20260806 17:07:10.278483 629 webserver.cc:460] Webserver started at http://127.9.111.125:37071/ using document root <none> and password file <none>
I20260806 17:07:10.279135 629 fs_manager.cc:359] Metadata directory not provided
I20260806 17:07:10.279211 629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:07:10.279486 629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:07:10.281955 629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-1/data/instance:
uuid: "fe1aea73e7fe405da7d99b24d6da9064"
format_stamp: "Formatted at 2026-08-06 17:07:10 on dist-test-slave-69rg"
server_key: "a9543585248d7e1db7b80cb880f34b08"
server_key_iv: "ec084e31d08909ce1a9cfd67b190c7c4"
server_key_version: "encryptionkey@0"
I20260806 17:07:10.282637 629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-1/wal/instance:
uuid: "fe1aea73e7fe405da7d99b24d6da9064"
format_stamp: "Formatted at 2026-08-06 17:07:10 on dist-test-slave-69rg"
server_key: "a9543585248d7e1db7b80cb880f34b08"
server_key_iv: "ec084e31d08909ce1a9cfd67b190c7c4"
server_key_version: "encryptionkey@0"
I20260806 17:07:10.287077 629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 17:07:10.290341 644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:07:10.292111 629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 17:07:10.292339 629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-1/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-1/wal
uuid: "fe1aea73e7fe405da7d99b24d6da9064"
format_stamp: "Formatted at 2026-08-06 17:07:10 on dist-test-slave-69rg"
server_key: "a9543585248d7e1db7b80cb880f34b08"
server_key_iv: "ec084e31d08909ce1a9cfd67b190c7c4"
server_key_version: "encryptionkey@0"
I20260806 17:07:10.292506 629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:07:10.313500 622 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } attempt: 3
I20260806 17:07:10.314584 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.314253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:32979 (local address 127.0.0.1:43570)
0806 17:07:10.314422 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.314500 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 17:07:10.315402 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.315145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:32979 (local address 127.0.0.1:43586)
0806 17:07:10.315276 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.315338 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.125:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 17:07:10.315575 622 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.125:32979: Network error: Client connection negotiation failed: client connection to 127.9.111.125:32979: connect: Connection refused (error 111)
I20260806 17:07:10.332142 629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:07:10.336486 629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:07:10.336759 629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:07:10.344846 629 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.125:32979
I20260806 17:07:10.345177 697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.125:32979 every 8 connection(s)
I20260806 17:07:10.346540 629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-1/data/info.pb
I20260806 17:07:10.350180 698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:07:10.351145 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 629
I20260806 17:07:10.351366 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-1/wal/instance
I20260806 17:07:10.351696 9661 external_mini_cluster.cc:1428] Setting key 837e1faf0ea754379d922692aad96122
I20260806 17:07:10.355293 698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.111.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.111.124@KRBTEST.COM" created.
Aug 06 17:07:10 dist-test-slave-69rg krb5kdc[540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036030, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.125@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:07:10.371805 626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.358001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53328 (local address 127.9.111.126:41975)
0806 17:07:10.358192 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.358196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.359460 (+ 1264us) server_negotiation.cc:407] Connection header received
0806 17:07:10.360512 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.360516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.360580 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.360688 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.362027 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.362573 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.363872 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.364122 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.364251 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.367839 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:10.367842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:10.367844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:10.367872 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:10.369461 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.370126 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.370130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.370131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.370216 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.370578 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.370582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.370583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.370791 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:10.370901 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:10.371471 (+ 570us) server_negotiation.cc:299] Negotiation successful
0806 17:07:10.371609 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 17:07:10.372653 701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.358589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:41975 (local address 127.0.0.1:53328)
0806 17:07:10.359264 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.359314 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 17:07:10.360286 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:07:10.360878 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:07:10.360884 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:07:10.361259 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:07:10.361860 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:10.361888 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.362734 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.362758 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:10.363731 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:10.363742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.364302 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.364307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:10.364499 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.365987 (+ 1488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:07:10.366006 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:07:10.367574 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:07:10.369601 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:10.369608 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:10.369627 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:10.370000 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:10.370312 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:10.370315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:10.370317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:10.370474 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:10.370899 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:07:10.370906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:07:10.371198 (+ 292us) client_negotiation.cc:769] Sending connection context
0806 17:07:10.371393 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 17:07:10.371752 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":213,"threads_started":1}
I20260806 17:07:10.374361 698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 17:07:10 dist-test-slave-69rg krb5kdc[540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036030, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.125@KRBTEST.COM for kudu/127.9.111.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.111.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.124.keytab.
Entry for principal kudu/127.9.111.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.124.keytab.
I20260806 17:07:10.386427 701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.375076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:32979 (local address 127.0.0.1:43588)
0806 17:07:10.375224 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.375247 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 17:07:10.375321 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:07:10.376103 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:07:10.376107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:07:10.376326 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:07:10.376414 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:10.376420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.377737 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.377740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:10.378388 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:10.378398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.379059 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.379063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:10.379150 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.379722 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:07:10.379736 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:07:10.381157 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:07:10.384425 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:10.384432 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:10.384437 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:10.384686 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:10.385154 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:10.385161 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:10.385162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:10.385255 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:10.385926 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:07:10.385929 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:07:10.386021 (+ 92us) client_negotiation.cc:769] Sending connection context
0806 17:07:10.386119 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 17:07:10.386264 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 17:07:10.386826 703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.375279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43588 (local address 127.9.111.125:32979)
0806 17:07:10.375698 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.375712 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.375733 (+ 21us) server_negotiation.cc:407] Connection header received
0806 17:07:10.375801 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.375805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.375970 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.376134 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.376532 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.377557 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.378518 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.378932 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.379057 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.381355 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:10.381369 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:10.381385 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:10.381446 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:10.384218 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.384839 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.384846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.384848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.384950 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.385357 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.385361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.385362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.385731 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:10.385943 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:10.386295 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 17:07:10.386501 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":144,"threads_started":1}
I20260806 17:07:10.387624 9661 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7cxihq/build/debug/bin/kudu
/tmp/dist-test-task7cxihq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.124:34763
--webserver_interface=127.9.111.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.124.keytab
--principal=kudu/127.9.111.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.111.84:41075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.111.126:41975,127.9.111.125:32979,127.9.111.124:34763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 17:07:10.389103 698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } has no permanent_uuid. Determining permanent_uuid...
I20260806 17:07:10.390399 701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.390028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:34763 (local address 127.0.0.1:58218)
0806 17:07:10.390221 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.390318 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:34763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260806 17:07:10.390753 646 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.111.124:34763: connect: Connection refused (error 111)
I20260806 17:07:10.391781 701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.391529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:34763 (local address 127.0.0.1:58234)
0806 17:07:10.391652 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.391708 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:34763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 17:07:10.392093 698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.124:34763: Network error: Client connection negotiation failed: client connection to 127.9.111.124:34763: connect: Connection refused (error 111)
I20260806 17:07:10.441393 698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } attempt: 1
I20260806 17:07:10.442585 701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.442147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:34763 (local address 127.0.0.1:58240)
0806 17:07:10.442423 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.442496 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:34763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260806 17:07:10.443470 701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.443216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:34763 (local address 127.0.0.1:58254)
0806 17:07:10.443350 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.443405 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:34763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 17:07:10.443632 698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.124:34763: Network error: Client connection negotiation failed: client connection to 127.9.111.124:34763: connect: Connection refused (error 111)
I20260806 17:07:10.458776 622 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } attempt: 4
Aug 06 17:07:10 dist-test-slave-69rg krb5kdc[540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036029, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for kudu/127.9.111.125@KRBTEST.COM
I20260806 17:07:10.470978 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.459524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:32979 (local address 127.0.0.1:43600)
0806 17:07:10.459750 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.459769 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 17:07:10.459880 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:07:10.460205 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:07:10.460207 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:07:10.460389 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:07:10.460517 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:10.460523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.461759 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.461762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:10.462902 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:10.462917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.463661 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.463664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:10.463760 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.464425 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:07:10.464439 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:07:10.465980 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:07:10.469212 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:10.469216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:10.469217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:10.469505 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:10.469920 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:10.469923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:10.469924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:10.470018 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:10.470501 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:07:10.470504 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:07:10.470601 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 17:07:10.470691 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 17:07:10.470831 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260806 17:07:10.471139 703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.459607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43600 (local address 127.9.111.125:32979)
0806 17:07:10.459764 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.459770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.459858 (+ 88us) server_negotiation.cc:407] Connection header received
0806 17:07:10.459978 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.459983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.460056 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.460166 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.460657 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.461572 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.463081 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.463506 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.463635 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.466173 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:10.466180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:10.466183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:10.466220 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:10.469040 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.469629 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.469633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.469635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.469742 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.470120 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.470124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.470126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.470400 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:10.470510 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:10.470801 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 17:07:10.470967 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 17:07:10.472048 622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } has no permanent_uuid. Determining permanent_uuid...
I20260806 17:07:10.473174 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.472889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:34763 (local address 127.0.0.1:58266)
0806 17:07:10.473036 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.473101 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:34763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 17:07:10.474058 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.473795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:34763 (local address 127.0.0.1:58280)
0806 17:07:10.473944 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.474001 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:34763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260806 17:07:10.474195 622 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.124:34763: Network error: Client connection negotiation failed: client connection to 127.9.111.124:34763: connect: Connection refused (error 111)
I20260806 17:07:10.511823 698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } attempt: 2
I20260806 17:07:10.512807 701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.512486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:34763 (local address 127.0.0.1:58296)
0806 17:07:10.512655 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.512724 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:34763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 17:07:10.513628 701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.513404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:34763 (local address 127.0.0.1:58306)
0806 17:07:10.513517 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.513572 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:34763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 17:07:10.513800 698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.124:34763: Network error: Client connection negotiation failed: client connection to 127.9.111.124:34763: connect: Connection refused (error 111)
W20260806 17:07:10.519071 704 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 17:07:10.519419 704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 17:07:10.519445 704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 17:07:10.519526 704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 17:07:10.524338 704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 17:07:10.524461 704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 17:07:10.524482 704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 17:07:10.524500 704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 17:07:10.524516 704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 17:07:10.531752 704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.111.84:41075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.111.126:41975,127.9.111.125:32979,127.9.111.124:34763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/krb5kdc/kudu_127.9.111.124.keytab
--principal=kudu/127.9.111.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.111.124:34763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.111.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 16:21:21 UTC on ccycloud.quasar-yoghow.root.comops.site
build id 4570853
I20260806 17:07:10.533082 704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 17:07:10.533406 622 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } attempt: 1
I20260806 17:07:10.534463 704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 17:07:10.534592 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.534220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:34763 (local address 127.0.0.1:58318)
0806 17:07:10.534405 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.534486 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:34763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 17:07:10.535387 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.535103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:34763 (local address 127.0.0.1:58326)
0806 17:07:10.535266 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.535323 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.111.124:34763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260806 17:07:10.535570 622 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.111.124:34763: Network error: Client connection negotiation failed: client connection to 127.9.111.124:34763: connect: Connection refused (error 111)
W20260806 17:07:10.543753 711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:07:10.543838 710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 17:07:10.543753 713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 17:07:10.543991 704 server_base.cc:1034] running on GCE node
I20260806 17:07:10.544755 704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 17:07:10.546023 704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 17:07:10.547214 704 hybrid_clock.cc:648] HybridClock initialized: now 1786036030547201 us; error 31 us; skew 500 ppm
Aug 06 17:07:10 dist-test-slave-69rg krb5kdc[540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036030, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 17:07:10.550160 704 init.cc:377] Logged in from keytab as kudu/127.9.111.124@KRBTEST.COM (short username kudu)
I20260806 17:07:10.551044 704 webserver.cc:460] Webserver started at http://127.9.111.124:43727/ using document root <none> and password file <none>
I20260806 17:07:10.551596 704 fs_manager.cc:359] Metadata directory not provided
I20260806 17:07:10.551662 704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 17:07:10.551837 704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 17:07:10.554086 704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-2/data/instance:
uuid: "27201d83dcd241289d0137e94301647c"
format_stamp: "Formatted at 2026-08-06 17:07:10 on dist-test-slave-69rg"
server_key: "ec0745f7fc0de19bf25eb3681b4c5c93"
server_key_iv: "8c0a4545d2a32b47067b9bad59153666"
server_key_version: "encryptionkey@0"
I20260806 17:07:10.554665 704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-2/wal/instance:
uuid: "27201d83dcd241289d0137e94301647c"
format_stamp: "Formatted at 2026-08-06 17:07:10 on dist-test-slave-69rg"
server_key: "ec0745f7fc0de19bf25eb3681b4c5c93"
server_key_iv: "8c0a4545d2a32b47067b9bad59153666"
server_key_version: "encryptionkey@0"
I20260806 17:07:10.558748 704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 17:07:10.561527 720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 17:07:10.563151 704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 17:07:10.563329 704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-2/data,/tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-2/wal
uuid: "27201d83dcd241289d0137e94301647c"
format_stamp: "Formatted at 2026-08-06 17:07:10 on dist-test-slave-69rg"
server_key: "ec0745f7fc0de19bf25eb3681b4c5c93"
server_key_iv: "8c0a4545d2a32b47067b9bad59153666"
server_key_version: "encryptionkey@0"
I20260806 17:07:10.563488 704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 17:07:10.590488 704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 17:07:10.594352 704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 17:07:10.594583 704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 17:07:10.602346 704 rpc_server.cc:307] RPC server started. Bound to: 127.9.111.124:34763
I20260806 17:07:10.602667 774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.111.124:34763 every 8 connection(s)
I20260806 17:07:10.602763 622 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } attempt: 2
I20260806 17:07:10.603641 704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-2/data/info.pb
I20260806 17:07:10.605007 698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } attempt: 3
I20260806 17:07:10.607084 775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 17:07:10.608093 9661 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7cxihq/build/debug/bin/kudu as pid 704
I20260806 17:07:10.608275 9661 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7cxihq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786035732873553-9661-0/minicluster-data/master-2/wal/instance
I20260806 17:07:10.608613 9661 external_mini_cluster.cc:1428] Setting key c62d6fddd627cbb1d8749942316676b9
Aug 06 17:07:10 dist-test-slave-69rg krb5kdc[540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036029, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.126@KRBTEST.COM for kudu/127.9.111.124@KRBTEST.COM
Aug 06 17:07:10 dist-test-slave-69rg krb5kdc[540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036030, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.125@KRBTEST.COM for kudu/127.9.111.124@KRBTEST.COM
I20260806 17:07:10.612418 775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } has no permanent_uuid. Determining permanent_uuid...
I20260806 17:07:10.618480 701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.605463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:34763 (local address 127.0.0.1:58342)
0806 17:07:10.605654 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.605669 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 17:07:10.605755 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:07:10.606379 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:07:10.606383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:07:10.606602 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:07:10.606729 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:10.606740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.608288 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.608292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:10.609140 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:10.609154 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.610246 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.610251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:10.610343 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.611030 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:07:10.611047 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:07:10.613018 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:07:10.615852 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:10.615856 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:10.615858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:10.616477 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:10.616990 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:10.616993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:10.616995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:10.617106 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:10.617935 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:07:10.617938 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:07:10.618044 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 17:07:10.618157 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 17:07:10.618304 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260806 17:07:10.618603 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.603347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:34763 (local address 127.0.0.1:58332)
0806 17:07:10.603533 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.603551 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 17:07:10.603649 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:07:10.606379 (+ 2730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:07:10.606383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:07:10.606601 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:07:10.606726 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:10.606734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.608305 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.608309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:10.609105 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:10.609123 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.610257 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.610264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:10.610364 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.611111 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:07:10.611124 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:07:10.612468 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:07:10.615852 (+ 3384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:10.615856 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:10.615858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:10.616124 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:10.617018 (+ 894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:10.617021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:10.617023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:10.617143 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:10.617954 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:07:10.617957 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:07:10.618085 (+ 128us) client_negotiation.cc:769] Sending connection context
0806 17:07:10.618212 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 17:07:10.618400 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 17:07:10.620426 778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.605541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58342 (local address 127.9.111.124:34763)
0806 17:07:10.605868 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.605877 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.605893 (+ 16us) server_negotiation.cc:407] Connection header received
0806 17:07:10.605931 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.605934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.606075 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.606870 (+ 795us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.607050 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.608125 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.609299 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.610093 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.610679 (+ 586us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.613194 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:10.613196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:10.613198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:10.613219 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:10.615714 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.616603 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.616619 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.616623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.616864 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.617230 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.617233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.617235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.617798 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:10.618715 (+ 917us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:10.618905 (+ 190us) server_negotiation.cc:299] Negotiation successful
0806 17:07:10.619337 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":258,"server-negotiator.queue_time_us":242,"spinlock_wait_cycles":13056,"thread_start_us":120,"threads_started":1}
I20260806 17:07:10.620434 777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.604180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58332 (local address 127.9.111.124:34763)
0806 17:07:10.604629 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.604655 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.604691 (+ 36us) server_negotiation.cc:407] Connection header received
0806 17:07:10.605587 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.605623 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.606076 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.606419 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.606875 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.608127 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.609277 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.610091 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.610684 (+ 593us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.612728 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:10.612737 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:10.612757 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:10.612792 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:10.615703 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.616570 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.616577 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.616583 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.616885 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.617270 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.617288 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.617290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.617805 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:10.618931 (+ 1126us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:10.619034 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 17:07:10.619674 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":526,"server-negotiator.queue_time_us":288,"spinlock_wait_cycles":20736,"thread_start_us":142,"threads_started":1}
Aug 06 17:07:10 dist-test-slave-69rg krb5kdc[540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.124@KRBTEST.COM
Aug 06 17:07:10 dist-test-slave-69rg krb5kdc[540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036030, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.124@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:07:10.626642 622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d: Bootstrap starting.
I20260806 17:07:10.627418 698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064: Bootstrap starting.
I20260806 17:07:10.628159 626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.615695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.9.111.126:41975)
0806 17:07:10.615828 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.615833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.616310 (+ 477us) server_negotiation.cc:407] Connection header received
0806 17:07:10.616491 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.616494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.616557 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.616635 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.617779 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.618540 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.620371 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.620760 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.620908 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.623980 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:10.623984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:10.623986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:10.624015 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:10.625873 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.626518 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.626522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.626524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.626612 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.626932 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.626935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.626936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.627215 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:10.627291 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:10.627821 (+ 530us) server_negotiation.cc:299] Negotiation successful
0806 17:07:10.627968 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 17:07:10.628741 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.614548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.126:41975 (local address 127.0.0.1:53340)
0806 17:07:10.616134 (+ 1586us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.616196 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 17:07:10.616352 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:07:10.616822 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:07:10.616828 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:07:10.617341 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:07:10.617637 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:10.617646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.618810 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.618813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:10.620212 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:10.620223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.620942 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.620946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:10.621197 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.621789 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:07:10.621803 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:07:10.623812 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:07:10.626025 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:10.626037 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:10.626042 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:10.626397 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:10.626737 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:10.626739 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:10.626741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:10.626829 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:10.627406 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:07:10.627412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:07:10.627590 (+ 178us) client_negotiation.cc:769] Sending connection context
0806 17:07:10.628354 (+ 764us) client_negotiation.cc:240] Negotiation successful
0806 17:07:10.628562 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1402,"mutex_wait_us":64,"spinlock_wait_cycles":3840,"thread_start_us":102,"threads_started":1}
I20260806 17:07:10.629024 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.609600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58356 (local address 127.9.111.124:34763)
0806 17:07:10.614071 (+ 4471us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.614081 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.614102 (+ 21us) server_negotiation.cc:407] Connection header received
0806 17:07:10.614134 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.614136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.614206 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.614301 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.615578 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.616445 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.617880 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.618266 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.618364 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.624205 (+ 5841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:10.624209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:10.624212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:10.624239 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:10.626246 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.626851 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.626855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.626857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.626954 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.627323 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.627327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.627329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.627605 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:10.627779 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:10.628678 (+ 899us) server_negotiation.cc:299] Negotiation successful
0806 17:07:10.628827 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":4367,"thread_start_us":115,"threads_started":1}
I20260806 17:07:10.629700 622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d: Neither blocks nor log segments found. Creating new log.
I20260806 17:07:10.630234 775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } has no permanent_uuid. Determining permanent_uuid...
I20260806 17:07:10.630357 698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064: Neither blocks nor log segments found. Creating new log.
I20260806 17:07:10.630659 622 log.cc:824] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d: Log is configured to *not* fsync() on all Append() calls
I20260806 17:07:10.631278 698 log.cc:824] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064: Log is configured to *not* fsync() on all Append() calls
I20260806 17:07:10.633915 622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d: No bootstrap required, opened a new log
I20260806 17:07:10.633915 698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064: No bootstrap required, opened a new log
Aug 06 17:07:10 dist-test-slave-69rg krb5kdc[540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036030, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.124@KRBTEST.COM for kudu/127.9.111.125@KRBTEST.COM
Aug 06 17:07:10 dist-test-slave-69rg krb5kdc[540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.126@KRBTEST.COM
I20260806 17:07:10.638916 698 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } } peers { permanent_uuid: "fe1aea73e7fe405da7d99b24d6da9064" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } } peers { permanent_uuid: "27201d83dcd241289d0137e94301647c" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } }
I20260806 17:07:10.639395 698 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1aea73e7fe405da7d99b24d6da9064, State: Initialized, Role: FOLLOWER
I20260806 17:07:10.639966 698 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } } peers { permanent_uuid: "fe1aea73e7fe405da7d99b24d6da9064" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } } peers { permanent_uuid: "27201d83dcd241289d0137e94301647c" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } }
I20260806 17:07:10.640127 622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } } peers { permanent_uuid: "fe1aea73e7fe405da7d99b24d6da9064" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } } peers { permanent_uuid: "27201d83dcd241289d0137e94301647c" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } }
I20260806 17:07:10.640448 622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39ab68f9d2e3499da77eeee0e4b01f2d, State: Initialized, Role: FOLLOWER
I20260806 17:07:10.640859 622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } } peers { permanent_uuid: "fe1aea73e7fe405da7d99b24d6da9064" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } } peers { permanent_uuid: "27201d83dcd241289d0137e94301647c" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } }
I20260806 17:07:10.641860 698 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:07:10.642338 703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.631224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.9.111.125:32979)
0806 17:07:10.631341 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.631345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.631575 (+ 230us) server_negotiation.cc:407] Connection header received
0806 17:07:10.631677 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.631684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.631746 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.631836 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.632370 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.633182 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.634413 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.634739 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.634816 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.638325 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:10.638330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:10.638332 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:10.638370 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:10.640207 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.640750 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.640754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.640756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.640844 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.641200 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.641206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.641207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.641519 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:10.641593 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:10.641990 (+ 397us) server_negotiation.cc:299] Negotiation successful
0806 17:07:10.642146 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 17:07:10.642385 622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:07:10.642649 626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.630848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53346 (local address 127.9.111.126:41975)
0806 17:07:10.630973 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.630977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.631179 (+ 202us) server_negotiation.cc:407] Connection header received
0806 17:07:10.631474 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.631477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.631536 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.631615 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.632516 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.633242 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.634392 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.634739 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.634816 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.638487 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:10.638492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:10.638495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:10.638526 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:10.640589 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.641141 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.641147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.641151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.641242 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.641568 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.641571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.641572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.641792 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:10.641980 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:10.642362 (+ 382us) server_negotiation.cc:299] Negotiation successful
0806 17:07:10.642492 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 17:07:10.642889 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.631172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.125:32979 (local address 127.0.0.1:43614)
0806 17:07:10.631479 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.631492 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 17:07:10.631578 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:07:10.631876 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:07:10.631880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:07:10.632064 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:07:10.632221 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:10.632229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.633334 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.633337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:10.634241 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:10.634251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.634982 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.634985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:10.635073 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.635761 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:07:10.635796 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:07:10.638070 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:07:10.640364 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:10.640368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:10.640370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:10.640630 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:10.640971 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:10.640975 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:10.640977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:10.641076 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:10.641672 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:07:10.641675 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:07:10.641775 (+ 100us) client_negotiation.cc:769] Sending connection context
0806 17:07:10.642550 (+ 775us) client_negotiation.cc:240] Negotiation successful
0806 17:07:10.642708 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":108,"threads_started":1}
I20260806 17:07:10.644068 775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } has no permanent_uuid. Determining permanent_uuid...
W20260806 17:07:10.649461 808 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:07:10.649751 808 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 06 17:07:10 dist-test-slave-69rg krb5kdc[540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036030, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.111.124@KRBTEST.COM for kudu/127.9.111.124@KRBTEST.COM
I20260806 17:07:10.651408 785 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } } peers { permanent_uuid: "fe1aea73e7fe405da7d99b24d6da9064" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } } peers { permanent_uuid: "27201d83dcd241289d0137e94301647c" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } } }
Aug 06 17:07:10 dist-test-slave-69rg krb5kdc[540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786036029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.111.125@KRBTEST.COM
I20260806 17:07:10.651679 785 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 17:07:10.654255 802 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260806 17:07:10.656170 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.645134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58370 (local address 127.9.111.124:34763)
0806 17:07:10.645265 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.645271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.645722 (+ 451us) server_negotiation.cc:407] Connection header received
0806 17:07:10.645921 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.645925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.645971 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.646023 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.646538 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.647268 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.648273 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.648659 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.648733 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.652281 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:10.652285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:10.652289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:10.652314 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:10.654369 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.654823 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.654827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.654830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.654911 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.655203 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.655208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.655213 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.655452 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:10.655527 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:10.655841 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 17:07:10.655983 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 17:07:10.656594 703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.645018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.9.111.125:32979)
0806 17:07:10.645324 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.645327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.645337 (+ 10us) server_negotiation.cc:407] Connection header received
0806 17:07:10.645367 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.645369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.645403 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.645496 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.646327 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.647116 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.648287 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.648651 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.648743 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.652323 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:10.652326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:10.652329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:10.652356 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:10.654306 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.654848 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.654851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.654853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.654932 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.655237 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.655242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.655247 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.655451 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:10.655640 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:10.656328 (+ 688us) server_negotiation.cc:299] Negotiation successful
0806 17:07:10.656446 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260806 17:07:10.656625 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.644807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.111.124:34763 (local address 127.0.0.1:58370)
0806 17:07:10.645659 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0806 17:07:10.645673 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 17:07:10.645782 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 17:07:10.646170 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 17:07:10.646172 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 17:07:10.646349 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0806 17:07:10.646431 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:10.646438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.647373 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.647375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:10.648156 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 17:07:10.648166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.648868 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.648871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 17:07:10.648945 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.649572 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 17:07:10.649584 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 17:07:10.652112 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 17:07:10.654473 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:10.654476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:10.654478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:10.654716 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:10.655017 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 17:07:10.655021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 17:07:10.655024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 17:07:10.655107 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 17:07:10.655587 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 17:07:10.655592 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 17:07:10.655678 (+ 86us) client_negotiation.cc:769] Sending connection context
0806 17:07:10.656365 (+ 687us) client_negotiation.cc:240] Negotiation successful
0806 17:07:10.656489 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779}
I20260806 17:07:10.660275 775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c: Bootstrap starting.
I20260806 17:07:10.661269 9661 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 17:07:10.655478 553 master_runner.cc:186] Error getting master registration for 127.9.111.124:34763: OK, instance_id { permanent_uuid: "27201d83dcd241289d0137e94301647c" instance_seqno: 1786036030600162 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260806 17:07:10.661880 802 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:07:10.657459 786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } } peers { permanent_uuid: "fe1aea73e7fe405da7d99b24d6da9064" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } } peers { permanent_uuid: "27201d83dcd241289d0137e94301647c" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } } }
I20260806 17:07:10.661964 553 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 17:07:10.662001 786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:07:10.662616 594 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53346
I20260806 17:07:10.662873 594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 17:07:10.662983 594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:07:10.663072 775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c: Neither blocks nor log segments found. Creating new log.
I20260806 17:07:10.663748 775 log.cc:824] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c: Log is configured to *not* fsync() on all Append() calls
I20260806 17:07:10.664503 594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } } peers { permanent_uuid: "fe1aea73e7fe405da7d99b24d6da9064" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } } peers { permanent_uuid: "27201d83dcd241289d0137e94301647c" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } }
I20260806 17:07:10.666142 775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c: No bootstrap required, opened a new log
I20260806 17:07:10.666456 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fe1aea73e7fe405da7d99b24d6da9064"
I20260806 17:07:10.666867 671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064 [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:07:10.668402 671 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39ab68f9d2e3499da77eeee0e4b01f2d in term 1.
I20260806 17:07:10.669162 572 leader_election.cc:302] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39ab68f9d2e3499da77eeee0e4b01f2d, fe1aea73e7fe405da7d99b24d6da9064; no voters:
I20260806 17:07:10.669286 594 leader_election.cc:288] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [CANDIDATE]: Term 1 election: Requested vote from peers fe1aea73e7fe405da7d99b24d6da9064 (127.9.111.125:32979), 27201d83dcd241289d0137e94301647c (127.9.111.124:34763)
I20260806 17:07:10.669520 786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 17:07:10.669800 786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [term 1 LEADER]: Becoming Leader. State: Replica: 39ab68f9d2e3499da77eeee0e4b01f2d, State: Running, Role: LEADER
I20260806 17:07:10.670253 786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } } peers { permanent_uuid: "fe1aea73e7fe405da7d99b24d6da9064" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } } peers { permanent_uuid: "27201d83dcd241289d0137e94301647c" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } }
I20260806 17:07:10.672822 812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39ab68f9d2e3499da77eeee0e4b01f2d. Latest consensus state: current_term: 1 leader_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } } peers { permanent_uuid: "fe1aea73e7fe405da7d99b24d6da9064" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } } peers { permanent_uuid: "27201d83dcd241289d0137e94301647c" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } } }
I20260806 17:07:10.672992 812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [sys.catalog]: This master's current role is: LEADER
I20260806 17:07:10.670089 629 master_runner.cc:186] Error getting master registration for 127.9.111.124:34763: OK, instance_id { permanent_uuid: "27201d83dcd241289d0137e94301647c" instance_seqno: 1786036030600162 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 17:07:10.673748 629 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 17:07:10.675891 775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } } peers { permanent_uuid: "fe1aea73e7fe405da7d99b24d6da9064" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } } peers { permanent_uuid: "27201d83dcd241289d0137e94301647c" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } }
I20260806 17:07:10.676247 775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27201d83dcd241289d0137e94301647c, State: Initialized, Role: FOLLOWER
I20260806 17:07:10.676640 775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } } peers { permanent_uuid: "fe1aea73e7fe405da7d99b24d6da9064" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } } peers { permanent_uuid: "27201d83dcd241289d0137e94301647c" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } }
I20260806 17:07:10.677239 822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } } peers { permanent_uuid: "fe1aea73e7fe405da7d99b24d6da9064" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } } peers { permanent_uuid: "27201d83dcd241289d0137e94301647c" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } } }
I20260806 17:07:10.677512 822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:07:10.678293 775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 17:07:10.678994 747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27201d83dcd241289d0137e94301647c"
I20260806 17:07:10.679266 747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c [term 0 FOLLOWER]: Advancing to term 1
I20260806 17:07:10.680907 747 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39ab68f9d2e3499da77eeee0e4b01f2d in term 1.
I20260806 17:07:10.682320 821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 17:07:10.683495 821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260806 17:07:10.685052 833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 17:07:10.685137 833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 17:07:10.687361 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.674005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58372 (local address 127.9.111.124:34763)
0806 17:07:10.674136 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.674141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.674157 (+ 16us) server_negotiation.cc:407] Connection header received
0806 17:07:10.674206 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.674209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.674264 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.674359 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.676104 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.676974 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.678330 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.678672 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.678794 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.680741 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:10.680745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:10.680748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:10.680784 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:10.682767 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.683362 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.683366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.683368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.683464 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.683818 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.683822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.683824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.684064 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:10.684190 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:10.686491 (+ 2301us) server_negotiation.cc:299] Negotiation successful
0806 17:07:10.687138 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 17:07:10.697594 626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.672068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.9.111.126:41975)
0806 17:07:10.672200 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.672205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.685033 (+ 12828us) server_negotiation.cc:407] Connection header received
0806 17:07:10.685209 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.685213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.685267 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.685332 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.687857 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.688607 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.690097 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.690470 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.690560 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.692849 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:10.692854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:10.692857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:10.692898 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:10.694871 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.695947 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.695952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.695954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.696083 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.696453 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.696457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.696459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.696721 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:10.696858 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:10.697254 (+ 396us) server_negotiation.cc:299] Negotiation successful
0806 17:07:10.697408 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 17:07:10.700124 747 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c [term 1 FOLLOWER]: Refusing update from remote peer 39ab68f9d2e3499da77eeee0e4b01f2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:07:10.700991 786 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "27201d83dcd241289d0137e94301647c" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:07:10.706185 822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39ab68f9d2e3499da77eeee0e4b01f2d. Latest consensus state: current_term: 1 leader_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } } peers { permanent_uuid: "fe1aea73e7fe405da7d99b24d6da9064" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } } peers { permanent_uuid: "27201d83dcd241289d0137e94301647c" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } } }
I20260806 17:07:10.706506 822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:07:10.707907 671 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064 [term 1 FOLLOWER]: Refusing update from remote peer 39ab68f9d2e3499da77eeee0e4b01f2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 17:07:10.708974 786 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe1aea73e7fe405da7d99b24d6da9064" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 17:07:10.716539 812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } } peers { permanent_uuid: "fe1aea73e7fe405da7d99b24d6da9064" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } } peers { permanent_uuid: "27201d83dcd241289d0137e94301647c" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } } }
I20260806 17:07:10.716728 812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [sys.catalog]: This master's current role is: LEADER
I20260806 17:07:10.716869 835 mvcc.cc:205] Tried to move back new op lower bound from 7315603581739114496 to 7315603581629554688. Current Snapshot: MvccSnapshot[applied={T|T < 7315603581739114496}]
I20260806 17:07:10.717089 822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } } peers { permanent_uuid: "fe1aea73e7fe405da7d99b24d6da9064" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } } peers { permanent_uuid: "27201d83dcd241289d0137e94301647c" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } } }
I20260806 17:07:10.717295 822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:07:10.718547 785 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39ab68f9d2e3499da77eeee0e4b01f2d. Latest consensus state: current_term: 1 leader_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } } peers { permanent_uuid: "fe1aea73e7fe405da7d99b24d6da9064" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } } peers { permanent_uuid: "27201d83dcd241289d0137e94301647c" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } } }
I20260806 17:07:10.718715 785 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:07:10.720317 703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:10.672816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43632 (local address 127.9.111.125:32979)
0806 17:07:10.672939 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.672944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.698436 (+ 25492us) server_negotiation.cc:407] Connection header received
0806 17:07:10.699713 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.699718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.699798 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.699857 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.700837 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.701640 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.702893 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:10.703250 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:10.703336 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:10.705852 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 17:07:10.705857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 17:07:10.705860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 17:07:10.705901 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 17:07:10.711417 (+ 5516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.718668 (+ 7251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.718673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.718674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.718813 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 17:07:10.719171 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 17:07:10.719175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 17:07:10.719177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 17:07:10.719427 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 17:07:10.719539 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:10.719963 (+ 424us) server_negotiation.cc:299] Negotiation successful
0806 17:07:10.720148 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 17:07:10.724449 836 mvcc.cc:205] Tried to move back new op lower bound from 7315603581739114496 to 7315603581629554688. Current Snapshot: MvccSnapshot[applied={T|T < 7315603581739114496}]
I20260806 17:07:10.732595 786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } } peers { permanent_uuid: "fe1aea73e7fe405da7d99b24d6da9064" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } } peers { permanent_uuid: "27201d83dcd241289d0137e94301647c" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } } }
I20260806 17:07:10.732761 786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d [sys.catalog]: This master's current role is: LEADER
I20260806 17:07:10.734187 785 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ab68f9d2e3499da77eeee0e4b01f2d" member_type: VOTER last_known_addr { host: "127.9.111.126" port: 41975 } } peers { permanent_uuid: "fe1aea73e7fe405da7d99b24d6da9064" member_type: VOTER last_known_addr { host: "127.9.111.125" port: 32979 } } peers { permanent_uuid: "27201d83dcd241289d0137e94301647c" member_type: VOTER last_known_addr { host: "127.9.111.124" port: 34763 } } }
I20260806 17:07:10.734421 785 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 17:07:10.734455 838 mvcc.cc:205] Tried to move back new op lower bound from 7315603581739114496 to 7315603581629554688. Current Snapshot: MvccSnapshot[applied={T|T < 7315603581739114496}]
I20260806 17:07:10.735679 821 catalog_manager.cc:1347] Generated new cluster ID: 895cc6f188bd411382bbea3ed77981f4
I20260806 17:07:10.735772 821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 17:07:10.783573 821 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 17:07:10.785125 821 catalog_manager.cc:1504] Loading token signing keys...
I20260806 17:07:10.802047 821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39ab68f9d2e3499da77eeee0e4b01f2d: Generated new TSK 0
I20260806 17:07:10.803035 821 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 17:07:10.842671 626 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:10.839233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53364 (local address 127.9.111.126:41975)
0806 17:07:10.839380 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.839386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.841096 (+ 1710us) server_negotiation.cc:407] Connection header received
0806 17:07:10.841607 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.841611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.841672 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.841729 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.842519 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53364: BlockingRecv error: recv got EOF from 127.0.0.1:53364 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 17:07:10.843875 703 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:10.840024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.9.111.125:32979)
0806 17:07:10.840136 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.840141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.841515 (+ 1374us) server_negotiation.cc:407] Connection header received
0806 17:07:10.843060 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.843063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.843116 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.843168 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.843755 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43642: BlockingRecv error: recv got EOF from 127.0.0.1:43642 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 17:07:10.845464 780 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:10.840795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58384 (local address 127.9.111.124:34763)
0806 17:07:10.840925 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.840930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.844487 (+ 3557us) server_negotiation.cc:407] Connection header received
0806 17:07:10.844709 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.844713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.844765 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.844817 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.845343 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58384: BlockingRecv error: recv got EOF from 127.0.0.1:58384 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260806 17:07:10.853466 703 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:10.850908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.9.111.125:32979)
0806 17:07:10.851044 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.851049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.851743 (+ 694us) server_negotiation.cc:407] Connection header received
0806 17:07:10.852199 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.852204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.852303 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.852388 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.853322 (+ 934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43648: BlockingRecv error: recv got EOF from 127.0.0.1:43648 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 17:07:10.859280 703 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:10.857389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.9.111.125:32979)
0806 17:07:10.857511 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.857516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.858226 (+ 710us) server_negotiation.cc:407] Connection header received
0806 17:07:10.858441 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.858444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.858501 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.858558 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.859162 (+ 604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43658: BlockingRecv error: recv got EOF from 127.0.0.1:43658 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 17:07:10.865468 703 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:10.863605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.9.111.125:32979)
0806 17:07:10.863724 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.863728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.864367 (+ 639us) server_negotiation.cc:407] Connection header received
0806 17:07:10.864595 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.864600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.864663 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.864721 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.865345 (+ 624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43674: BlockingRecv error: recv got EOF from 127.0.0.1:43674 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260806 17:07:10.873852 703 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:10.871880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43684 (local address 127.9.111.125:32979)
0806 17:07:10.872048 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.872053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.872680 (+ 627us) server_negotiation.cc:407] Connection header received
0806 17:07:10.872970 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.872974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.873030 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.873086 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.873672 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43684: BlockingRecv error: recv got EOF from 127.0.0.1:43684 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260806 17:07:10.886574 703 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:10.884527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43686 (local address 127.9.111.125:32979)
0806 17:07:10.884669 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.884675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.885377 (+ 702us) server_negotiation.cc:407] Connection header received
0806 17:07:10.885615 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.885619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.885679 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.885740 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.886446 (+ 706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43686: BlockingRecv error: recv got EOF from 127.0.0.1:43686 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 17:07:10.907116 703 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:10.905280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43700 (local address 127.9.111.125:32979)
0806 17:07:10.905411 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.905416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.906068 (+ 652us) server_negotiation.cc:407] Connection header received
0806 17:07:10.906277 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.906281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.906341 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.906394 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.906974 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43700: BlockingRecv error: recv got EOF from 127.0.0.1:43700 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 17:07:10.943249 703 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:10.941729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43708 (local address 127.9.111.125:32979)
0806 17:07:10.941909 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:10.941914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:10.942142 (+ 228us) server_negotiation.cc:407] Connection header received
0806 17:07:10.942362 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:10.942365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:10.942421 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:10.942551 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:10.943061 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43708: BlockingRecv error: recv got EOF from 127.0.0.1:43708 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 17:07:11.011803 703 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:11.009963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43716 (local address 127.9.111.125:32979)
0806 17:07:11.010134 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:11.010138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:11.010238 (+ 100us) server_negotiation.cc:407] Connection header received
0806 17:07:11.010560 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:11.010570 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:11.010662 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:11.010757 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:11.011589 (+ 832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43716: BlockingRecv error: recv got EOF from 127.0.0.1:43716 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260806 17:07:11.144392 703 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:11.142454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.9.111.125:32979)
0806 17:07:11.142616 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:11.142621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:11.143081 (+ 460us) server_negotiation.cc:407] Connection header received
0806 17:07:11.143395 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:11.143398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:11.143482 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:11.143575 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:11.144247 (+ 672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43720: BlockingRecv error: recv got EOF from 127.0.0.1:43720 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 17:07:11.404712 703 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 17:07:11.403238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.9.111.125:32979)
0806 17:07:11.403436 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:11.403442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:11.403593 (+ 151us) server_negotiation.cc:407] Connection header received
0806 17:07:11.403845 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:11.403848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:11.403898 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:11.403999 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 17:07:11.404590 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43734: BlockingRecv error: recv got EOF from 127.0.0.1:43734 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20260806 17:07:11.651310 808 catalog_manager.cc:1259] Loaded cluster ID: 895cc6f188bd411382bbea3ed77981f4
I20260806 17:07:11.651434 808 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064: loading cluster ID for follower catalog manager: success
I20260806 17:07:11.653151 808 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064: acquiring CA information for follower catalog manager: success
I20260806 17:07:11.654469 808 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fe1aea73e7fe405da7d99b24d6da9064: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 17:07:11.686676 833 catalog_manager.cc:1259] Loaded cluster ID: 895cc6f188bd411382bbea3ed77981f4
I20260806 17:07:11.686791 833 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c: loading cluster ID for follower catalog manager: success
I20260806 17:07:11.688483 833 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c: acquiring CA information for follower catalog manager: success
I20260806 17:07:11.689998 833 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 27201d83dcd241289d0137e94301647c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 17:07:11.923887 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:11.919488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43748 (local address 127.9.111.125:32979)
0806 17:07:11.919813 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:11.919818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:11.919835 (+ 17us) server_negotiation.cc:407] Connection header received
0806 17:07:11.919984 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:11.919987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:11.920201 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:11.920357 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:07:11.920647 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:11.921200 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:11.922218 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:11.922487 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:11.922615 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:11.922979 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:07:11.923435 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:07:11.923578 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:11.923641 (+ 63us) server_negotiation.cc:299] Negotiation successful
0806 17:07:11.923705 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":108,"threads_started":1}
I20260806 17:07:11.932062 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:11.927560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58390 (local address 127.9.111.124:34763)
0806 17:07:11.927880 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:11.927884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:11.927899 (+ 15us) server_negotiation.cc:407] Connection header received
0806 17:07:11.927984 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:11.927986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:11.928109 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:11.928224 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:07:11.928638 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:11.929228 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:11.930340 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:11.930627 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:11.930737 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:11.931086 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:07:11.931560 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:07:11.931663 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:11.931791 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 17:07:11.931851 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
I20260806 17:07:11.939877 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:11.935926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43764 (local address 127.9.111.125:32979)
0806 17:07:11.936050 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:11.936054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:11.936357 (+ 303us) server_negotiation.cc:407] Connection header received
0806 17:07:11.936451 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:11.936453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:11.936509 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:11.936581 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:07:11.937075 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:11.937595 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:11.938736 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:11.938975 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:11.939090 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:11.939363 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:07:11.939446 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:07:11.939526 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:11.939641 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 17:07:11.939699 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 17:07:11.940605 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:11.935524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58398 (local address 127.9.111.124:34763)
0806 17:07:11.935665 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:11.935672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:11.936255 (+ 583us) server_negotiation.cc:407] Connection header received
0806 17:07:11.936450 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:11.936453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:11.936513 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:11.936562 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:07:11.937053 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:11.937777 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:11.939124 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:11.939480 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:11.939561 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:11.940124 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:07:11.940204 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:07:11.940263 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:11.940425 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 17:07:11.940484 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 17:07:11.941082 922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 17:07:11.935516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53368 (local address 127.9.111.126:41975)
0806 17:07:11.935968 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0806 17:07:11.935974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 17:07:11.935990 (+ 16us) server_negotiation.cc:407] Connection header received
0806 17:07:11.936220 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 17:07:11.936223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 17:07:11.936408 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 17:07:11.936535 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 17:07:11.936773 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:11.937562 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:11.939125 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 17:07:11.939480 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 17:07:11.939572 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 17:07:11.940186 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 17:07:11.940569 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 17:07:11.940660 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 17:07:11.940822 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 17:07:11.940894 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":149,"threads_started":1}
I20260806 17:07:11.945106 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 553
I20260806 17:07:11.952628 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 629
I20260806 17:07:11.959887 9661 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7cxihq/build/debug/bin/kudu with pid 704
2026-08-06T17:07:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3356 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6586 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (299074 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST